Managing file objects in a data storage system

ABSTRACT

Managing file objects in a data storage system includes providing a plurality of metadata servers, each metadata server having information for only a subset of file objects in the storage system, each subset for each of the metadata servers being less than a total number of file objects for the system and includes providing at least one metadata location server, where an entity accessing a specific file object of the system determines which of the metadata servers contains data for the specific file object by first accessing the at least one metadata location server. Information for the subset of file objects may include a table having file identifiers and metadata location information. The metadata location information may point to data storage for a metadata object for a file.

BACKGROUND OF THE INVENTION

1. Technical Field

This application relates to the field of storing data, and moreparticularly to the field of data storage services in a scalable highcapacity system.

2. Description of Related Art

It has been estimated that the amount of digital information created,captured, and replicated in 2006 was 161 exabytes or 161 billiongigabytes, which is about three million times the information in all thebooks ever written. It is predicted that between 2006 and 2010, theinformation added annually to the digital universe will increase morethan six fold from 161 exabytes to 988 exabytes. The type of informationresponsible for this massive growth is rich digital media andunstructured business content. There is also an ongoing conversion fromanalog to digital formats—film to digital image capture, analog todigital voice, and analog to digital TV.

The rich digital media and unstructured business content have uniquecharacteristics and storage requirements that are different thanstructured data types (e.g. database records), for which many of today'sstorage systems were specially designed. Many conventional storagesystems are highly optimized to deliver high performance I/O for smallchunks of data. Furthermore, these systems were designed to supportgigabyte and terabyte sized information stores.

In contrast, rich digital media and unstructured business content havegreater capacity requirements (petabyte versus gigabyte/terabyte sizedsystems), less predictable growth and access patterns, large file sizes,billions and billions of objects, high throughput requirements, singlewriter, multiple reader access patterns, and a need for multi-platformaccessibility. Conventional storage systems have met these needs in partby using specialized hardware platforms to achieve required levels ofperformance and reliability. Unfortunately, the use of specializedhardware results in higher customer prices and may not support volumeeconomics as the capacity demands grow large—a differentiatingcharacteristic of rich digital media and unstructured business content.

Some of the cost issues have been addressed with tiered storage, whichattempts to reduce the capital and operational costs associated withkeeping all information on a single high-cost storage tier. However,tiered storage comes with a complex set of decisions surroundingtechnology, data durability, functionality and even storage vendor.Tiered storage solutions may introduce unrelated platforms,technologies, and software titles having non-zero operational costs andmanagement requirements that become strained as the quantity of dataincreases.

In addition, tiered storage may cause a data replica incoherence whichresults in multiple, disjoint copies of information existing across thetiers of storage. For example, storage management software handling databackup and recovery may make multiple copies of information sets on eachstorage tier (e.g. snapshots, backup sets, etc). Information Life-cycleManagement (ILM) software dealing with information migration from onetier to another may create additional and often overlapping copies ofthe data. Replication software may make an extra copy of the informationset within a particular tier in order to increase performance toaccessing applications. Each of these functions typically runsautonomously from one another. The software may be unable to realizeand/or take advantage of the multiple replicas of the same informationset.

In addition, for large scale unstructured information stores, it may bedifficult to maintain a system and manage the environment as componentsfail. For example, a two petabyte information store may be comprised ofeight thousand 250-gigabyte disk drives. Disk failures should be handledin a different manner in a system of this scale so that the systemcontinues to operate relatively smoothly whenever one or only a few ofthe disk drives fail.

Thus, it would be desirable to provide a storage system that addressesdifficulties associated with high-cost specialized hardware, storagetiering, and failure management.

SUMMARY OF THE INVENTION

According to the system described herein, managing file objects in adata storage system includes providing a plurality of metadata servers,each metadata server having information for only a subset of fileobjects in the storage system, each subset for each of the metadataservers being less than a total number of file objects for the systemand includes providing at least one metadata location server, where anentity accessing a specific file object of the system determines whichof the metadata servers contains data for the specific file object byfirst accessing the at least one metadata location server. Informationfor the subset of file objects may include a table having fileidentifiers and metadata location information. The metadata locationinformation may point to data storage for a metadata object for a file.The table may also include lease information that indicates entitiesthat have read permission for the file object and entities that havewrite permission for the file object. File objects may be provided withunique object identifiers. The at least one metadata location server maydetermine an appropriate metadata server to use for a file objectaccording to an object identification value for the file object. Atleast some of the metadata servers may maintain a table that correlatesa subset of the object identification values with file objects. At leastsome of the metadata servers may be provided with a unique subset ofobject identification values for assigning to new file objects.

According further to the system described herein, computer software,provided in a computer-readable medium, manages file objects in a datastorage system. The software includes executable code, provided on aplurality of metadata servers, that maintains information for only asubset of file objects in the storage system, each subset for each ofthe metadata servers being less than a total number of file objects forthe system and includes executable code, provided on at least onemetadata location server, that provides an indication of which of themetadata servers contains data for a specific file object in response toa request therefor provided to the at least one metadata server.Information for the subset of file objects may include a table havingfile identifiers and metadata location information. The metadatalocation information may point to data storage for a metadata object fora file. The table may also includes lease information that indicatesentities that have read permission for the file object and entities thathave write permission for the file object. File objects may be providedwith unique object identifiers. The at least one metadata locationserver may include executable code that determines an appropriatemetadata server to use for a file object according to an objectidentification value for the file object. At least some of the metadataservers may maintain a table that correlates a subset of the objectidentification values with file objects. At least some of the metadataservers may be provided with a unique subset of object identificationvalues for assigning to new file objects.

According further to the system described herein, a data storage systemincludes a plurality of clients that access file objects of the storagesystem and a plurality of interconnected servers coupled to the clients,the servers including a plurality of metadata servers, each metadataserver having information for only a subset of file objects in thestorage system, each subset for each of the metadata servers being lessthan a total number of file objects for the system, the servers alsoincluding at least one metadata location server, where a clientaccessing a specific file object of the system determines which of themetadata servers contains data for the specific file object by firstaccessing the at least one metadata location server. Information for thesubset of file objects may include a table having file identifiers andmetadata location information. The metadata location information maypoint to data storage for a metadata object for a file. File objects maybe provided with unique object identifiers. The at least one metadatalocation server may determine an appropriate metadata server to use fora file object according to an object identification value for the fileobjects for the file. At least some of the metadata servers may maintaina table that correlates a subset of the object identification valueswith file objects. At least some of the metadata servers may be providedwith a unique subset of object identification values for assigning tonew file objects.

According further to the system described herein, managing file objectsin a data storage system includes providing a plurality of a first typeof server, each having information for only a subset of file objects inthe storage system and includes providing at least one of a second typeof server that provides information indicating which of the servers ofthe first type contains data for a specific file object by firstaccessing the at least one server of the second type.

According further to the system described herein, a data storage systemincludes a plurality of clients that access file objects of the storagesystem and includes a plurality of interconnected servers coupled to theclients, the servers including a plurality of metadata servers, eachmetadata server having information for only a subset of file objects inthe storage system, each subset for each of the metadata servers beingless than a total number of file objects for the system, the serversalso including at least one metadata location server, where a clientaccessing a specific file of the system determines which of the metadataservers contains data for the specific file object by first accessingthe at least one metadata location server.

According further to the system described herein, managing a pluralityof files includes providing at least one server having a table withplurality of file identifiers, each of the file identifiers including apointer to a metadata file object containing a layout storage objectthat indicates storage parameters for corresponding files and includesaccessing a particular one of the files by obtaining information fromthe at least one server to access the layout storage objectcorresponding to the particular file to determine synchronous andasynchronous mirrors for the file. Each layout storage object may beprovided as a tree and leaves of the tree may map logical addresses of acorresponding file to data storage locations. At least one internal nodeof the tree may indicate that at least one leaf node thereof is a mirrorof an other leaf node thereof. In response to a data write and aninternal node indicating that a first leaf node thereof is anasynchronous mirror of a second leaf node thereof, data may be writtento the first leaf node and an update message may be provided to the atleast one server. Managing a plurality of files may also include the atleast one server placing information on a queue corresponding to themessage and servicing the queue to write the data to the second leafnode. In response to a data write and an internal node indicating that afirst leaf node thereof is a synchronous mirror of a second leaf nodethereof, data may be written to the first leaf node and to the secondleaf node. In response to accessing data being unsuccessful, a mirrorthereof may be accessed. The layout storage object may include pointersto data file objects that map a logical address space for acorresponding file to data storage space corresponding to physical datastorage. Each data file object may have a state indicating that the datafile object is one of: current, an out-of-date mirror of another dataobject, immutable, and invalid. In response to an attempt to write to adata file object that is immutable, new storage space may be allocatedand data may be written to the new storage space. In response to anattempt to read a data file object that is invalid, zeros may bereturned.

According further to the system described herein, computer software,provided in a computer-readable medium, manages a plurality of files.The software includes executable code provided in at least one serverthat manages a table with plurality of file identifiers, each of thefile identifiers including a pointer to a metadata file objectcontaining a layout storage object that indicates storage parameters forcorresponding files and includes executable code that accesses aparticular one of the files by obtaining information from the at leastone server to access the layout storage object corresponding to theparticular file to determine synchronous and asynchronous mirrors forthe file. Each layout storage object may be provided as a tree andleaves of the tree may map logical addresses of a corresponding file todata storage locations. At least one internal node of the tree mayindicate that at least one leaf node thereof is a mirror of an otherleaf node thereof. In response to a data write and an internal nodeindicating that a first leaf node thereof is an asynchronous mirror of asecond leaf node thereof, executable code may cause the data to bewritten to the first leaf node and an update message to be provided tothe at least one server. Executable code at the at least one server mayplace information on a queue corresponding to the message and mayservice the queue to write the data to the second leaf node. In responseto a data write and an internal node indicating that a first leaf nodethereof is a synchronous mirror of a second leaf node thereof,executable code may cause the data to be written to the first leaf nodeand to the second leaf node. The computer software may also includeexecutable code that, in response to accessing data being unsuccessful,accesses a mirror thereof. The layout storage object may includepointers to data file objects that map a logical address space for acorresponding file to data storage space corresponding to physical datastorage. Each data file object may have a state indicating that the datafile object is one of: current, an out-of-date mirror of another dataobject, immutable, and invalid. The computer software may also includeexecutable code that allocates new storage space and writes data to thenew storage space in response to an attempt to write to a data fileobject that is immutable. The computer software may also includeexecutable code that causes zeros to be returned in response to anattempt to read a data file object that is invalid.

According further to the system described herein, managing a pluralityof files includes providing at least one server having a plurality offile identifiers, each of the file identifiers including a pointer to ametadata file object containing a layout storage object that indicatesstorage parameters for corresponding files and includes accessing aparticular one of the files by obtaining information from the at leastone server to access the layout storage object corresponding to theparticular file to determine how data for the file is stored andretrieved.

According further to the system described herein, a data storage systemincludes a plurality of servers, where at least one server has a tablewith plurality of file identifiers, each of the file identifiersincluding a pointer to a metadata file object containing a layoutstorage object that indicates storage parameters for corresponding filesand includes a plurality of clients that access a particular one of thefiles by obtaining information from the at least one server to accessthe layout storage object corresponding to the particular file todetermine synchronous and asynchronous mirrors for the file.

According further to the system described herein, a data storage systemincludes a plurality of servers, where at least one server has aplurality of file identifiers, each of the file identifiers including apointer to a metadata file object containing a layout storage objectthat indicates storage parameters for corresponding files and includes aplurality of clients that access a particular one of the files byobtaining information from the at least one server to access the layoutstorage object corresponding to the particular file to determine howdata for the file is stored and retrieved.

According further to the system described herein, managing filesincludes receiving a plurality of policies for a new file, wheredifferent policies may be applied according to predetermined conditions,receiving a request to create a new file, selecting an appropriatepolicy for the new file according to the predetermined conditions,determining if sufficient resources are available according to theappropriate policy, and obtaining file objects to create the file inresponse to their being sufficient resources according to theappropriate policy. Policies may be received via a user managementinterface. The predetermined conditions may include an identity of aclient creating a file. The appropriate policy may include informationregarding mirrors for the file. The appropriate policy may indicate aminimum geographic distance between the mirrors. Determining ifsufficient resources are available may include consulting a resourcemanager. The resource manager may receive information about systemresources and maintains a table indicative thereof. The system resourcesmay include data storage areas.

According further to the system described herein, computer software,provided in a computer-readable medium, manages files. The softwareincludes executable code that receives a plurality of policies for a newfile, wherein different policies may be applied according topredetermined conditions, executable code that receives a request tocreate a new file, executable code that selects an appropriate policyfor the new file according to the predetermined conditions, executablecode that determines if sufficient resources are available according tothe appropriate policy and executable code that obtains file objects tocreate the file in response to their being sufficient resourcesaccording to the appropriate policy. Policies may be received via a usermanagement interface. The predetermined conditions may include anidentity of a client creating a file. The appropriate policy may includeinformation regarding mirrors for the file. The appropriate policy mayindicate a minimum geographic distance between the mirrors. Executablecode that determines if sufficient resources are available may includeexecutable code that consults a resource manager. The resource managermay include executable code that receives information about systemresources and maintains a table indicative thereof. The system resourcesmay include data storage areas.

According further to the system described herein, a data storage systemincludes a plurality of clients that access files and includes aplurality of interconnected servers, coupled to the clients, a subset ofthe servers receiving a plurality of policies for a new file, wheredifferent policies may be applied according to predetermined conditions,a subset of the servers receiving a request to create a new file, asubset of the servers selecting an appropriate policy for the new fileaccording to the predetermined conditions, a subset of the serversdetermining if sufficient resources are available according to theappropriate policy, and a subset of the servers obtaining file objectsto create the file in response to their being sufficient resourcesaccording to the appropriate policy. Policies may be received via a usermanagement interface. The predetermined conditions may include anidentity of a client creating a file. The appropriate policy may includeinformation regarding mirrors for the file. The servers include a subsetof servers that are consulted to determine if sufficient resources areavailable. The subset of servers that are consulted to determine ifsufficient resources are available may receive information about systemresources and maintains a table indicative thereof. The system resourcesmay include data storage areas.

According further to the system described herein, managing file objectsincludes receiving a plurality of policies for file objects, wheredifferent policies may be applied according to predetermined conditions,receiving a request to obtain new file objects, selecting an appropriatepolicy for the new file objects according to the predeterminedconditions, determining if sufficient resources are available accordingto the appropriate policy, and obtaining file objects in response totheir being sufficient resources according to the appropriate policy.

According further to the system described herein, a data storage systemincludes a plurality of clients that access file objects and includes aplurality of interconnected servers, coupled to the clients, a subset ofthe servers receiving a plurality of policies for new file objects,where different policies may be applied according to predeterminedconditions, a subset of the servers receiving a request to obtain newfile objects, a subset of the servers selecting an appropriate policyfor the new file objects according to the predetermined conditions, asubset of the servers determining if sufficient resources are availableaccording to the appropriate policy, and a subset of the serversobtaining the file objects in response to their being sufficientresources according to the appropriate policy.

According further to the system described herein, tracking storageresources includes providing a table containing storage resources alongwith capabilities and statuses thereof, updating the table in responseto a change of status of a storage resource, updating the table inresponse to a change in capabilities of a storage resource and, inresponse to an inquiry for a storage resource having a particularcapability, searching the table for a storage resource having theparticular capability. Tracking storage resources may also includeadding an element to the table in response to a new resource being addedto the system. The capabilities may include RAID striping, datadeduplication, and green operation. The status may be one of: on-line,off-line, and full. The storage resources may be disk drives. The diskdrives may be managed by data storage servers that present an OSDinterface for the disk drives. The table may be maintained by a resourcemanager server that receives information about storage resources fromother servers.

According further to the system described herein, computer software,provided in a computer-readable storage medium, tracks storageresources. The software includes a table that contains storage resourcesalong with capabilities and statuses thereof, executable code thatupdates the table in response to a change of status of a storageresource, executable code that updates the table in response to a changein capabilities of a storage resource, and executable code that searchesthe table for a storage resource having a particular capability inresponse to an inquiry for a storage resource having the particularcapability. The software may also include executable code that adds anelement to the table in response to a new resource being added to thesystem. The capabilities may include RAID striping, data deduplication,and green operation. The status may be one of: on-line, off-line, andfull. The storage resources may be disk drives. The disk drives may bemanaged by data storage servers that present an OSD interface for thedisk drives. The table may be maintained by a resource manager serverthat receives information about storage resources from other servers.

According further to the system described herein, a resource managerthat manages storage resources for a storage system includes aprocessing device and a computer-readable memory coupled to theprocessing device, the computer-readable memory having a table providedin a data structure and containing storage resources along withcapabilities and statuses thereof, the computer-readable memory alsohaving executable code that updates the table in response to a change ofstatus of a storage resource, executable code that updates the table inresponse to a change in capabilities of a storage resource, andexecutable code that searches the table for a storage resource having aparticular capability in response to an inquiry for a storage resourcehaving the particular capability. The computer-readable memory may alsocontain executable code that adds an element to the table in response toa new resource being added to the system. The capabilities may includeRAID striping, data deduplication, and green operation. The status maybe one of: on-line, off-line, and full. The storage resources may bedisk drives. The disk drives may be managed by data storage servers thatpresent an OSD interface for the disk drives.

According further to the system described herein, a data storage systemincludes a plurality of clients and a plurality of servers coupled tothe clients, where a subset of the servers manage storage resourcesusing a table containing storage resources along with capabilities andstatuses thereof, where the subset updates the table in response to achange of status of a storage resource, updates the table in response toa change in capabilities of a storage resource, and searches the tablefor a storage resource having the particular capability in response toan inquiry for a storage resource having a particular capability. Thesubset of servers may add an element to the table in response to a newresource being added to the system. The storage resources may be diskdrives.

According further to the system described herein, providing informationto a resource manager of a data storage system includes providinginformation to the resource manager in response to a change incapabilities of a storage resource, providing information to theresource manager in response to a change in status of a storageresource, and providing information to the resource manager in responseto adding a new storage resource. The storage resources may be diskdrives.

According further to the system described herein, accessing data fileobjects includes providing a file system interface on at least oneclient, where the file system interface allows an application running onthe at least one client to make file system calls to access the datafile objects and providing a Web Services interface on the at least oneclient, where the Web interface allows an application to access fileobjects using the Web at the same time that file objects are beingaccessed through the file system interface. The Web Services interfacemay be a SOAP interface and/or a REST interface. Accessing data fileobjects may also include providing a direct file object interface on theat least one client, where the direct file object interface allows anapplication to directly access file objects at the same time that fileobjects are being accessed using the Web Services interface and usingthe file system interface. Accessing data file objects may also includeproviding file name services that translate file names into file objectidentifiers, where the file name services are accessed by the filesystem interface and/or the Web Services interface. Providing the filesystem interface may include providing a virtual file system in a kerneladdress space of the at least one client. Accessing data file objectsmay also include providing a layout manager that manages file objects.The layout manager may be provided in the kernel address space of the atleast one client. The layout manager is provided in user address spaceof the at least one client. The virtual file system may access thelayout manager using a bridge between kernel memory address space anduser memory address space.

According further to the system described herein, a client that accessesa plurality of file objects includes a file system interface that allowsan application running on the at least one client to make file systemcalls to access the data file objects, a Web Services interface thatallows an application to access file objects using the Web at the sametime that file objects are being accessed through the file systeminterface, and a communication interface, coupled to the file systeminterface and to the Web Services interface, that exchanges file objectdata between the client and a plurality of servers containing the fileobjects. The client may also include a direct file object interface thatallows an application to directly access file objects at the same timethat file objects are being accessed using the Web Services interfaceand using the file system interface. The client may also include a filename service that translates file names into file object identifiers,where the file name service is accessed by the file system interfaceand/or the Web Services interface. The file system interface may includea virtual file system in a kernel address space of the at least oneclient. The client may also include a layout manager that manages fileobjects. The layout manager may be provided in the kernel address spaceof the at least one client. The layout manager may be provided in useraddress space of the at least one client. The virtual file system mayaccess the layout manager using a bridge between kernel memory addressspace and user memory address space.

According further to the system described herein, computer software,provided in a computer-readable storage medium, includes executable codethat allows an application running on at least one client to make filesystem calls to access the data file objects and includes executablecode that allows an application to access file objects using the Web atthe same time that file objects are being accessed through the filesystem interface. The computer software may also include executable codethat allows an application to directly access file objects at the sametime that file objects are being accessed using the Web and using filesystem calls. The computer software may also include executable codethat translates file names into file object identifiers.

According further to the system described herein, a systemsimultaneously allows accessing a plurality of file objects using a filesystem interface and a Web Services interface. The system includes aplurality of servers that contain the file objects and at least oneclient, coupled to the servers. The client includes a file systeminterface that allows an application running on the at least one clientto make file system calls to access the data file objects, a WebServices interface that allows an application to access file objectsusing the Web at the same time that file objects are being accessedthrough the file system interface, and a communication interface,coupled to the file system interface and the Web Services interface,that exchanges file object data between the client and a plurality ofservers containing the file objects. The file system interface mayinclude a virtual file system in a kernel address space of the client.

According further to the system described herein, managing objects for adata file includes obtaining a metadata object for the file, determiningif the metadata object indicates that a portion of the data fileincludes synchronous mirrors, and, in response to a write of new data tothe data file, writing the new data to each synchronous mirror. Dataobjects of a data file may be in one of four states: current, stale,empty, and immutable. In response to a read operation and thecorresponding data object being in a stale state, data may be read for acorresponding synchronous mirror that is in a current state.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating servers and clients according to anembodiment of the system described herein.

FIGS. 2A and 2B are diagrams illustrating a client coupled to serversand to other network(s) according to an embodiment of the systemdescribed herein.

FIG. 3 is a diagram illustrating a client having server operationssoftware, client software, and a plurality of interfaces therebetweenaccording to an embodiment of the system described herein.

FIG. 4 is a diagram illustrating a file having a metadata file objectand a plurality of data file objects according to an embodiment of thesystem described herein.

FIG. 5 is a diagram illustrating a metadata file object for a fileaccording to an embodiment of the system described herein.

FIG. 6 is a diagram illustrating an example of a layout storage objecttree for a file according to an embodiment of the system describedherein.

FIG. 7 is a diagram illustrating an example of a layout storage objecttree with multiple maps for a file according to an embodiment of thesystem described herein.

FIG. 8 is a diagram illustrating another example of a layout storageobject tree with multiple maps and replication nodes for a fileaccording to an embodiment of the system described herein.

FIG. 9 is a flowchart illustrating a client obtaining a lease for andoperating on a file according to an embodiment of the system describedherein.

FIG. 10 is a flowchart illustrating a client reading data from a fileaccording to an embodiment of the system described herein.

FIG. 11 is a flowchart illustrating a client writing data to a fileaccording to an embodiment of the system described herein.

FIG. 12 is a flowchart illustrating steps performed by a client inconnection with finding an alternative copy of data according to anembodiment of the system described herein.

FIG. 13 is a flowchart illustrating a client writing to synchronousmirrors for data according to an embodiment of the system describedherein.

FIG. 14 is a flow chart illustrating a client converting file names toobject identifiers according to an embodiment of the system describedherein.

FIG. 15 is a diagram illustrating a client having an application in usermemory address space and a having a VFS, file name services, kernel I/Odrivers, layout manager, and a communication interface in kernel memoryaddress space according to an embodiment of the system described herein.

FIG. 16 is a flow chart illustrating operation of a VFS at a clientaccording to an embodiment of the system described herein.

FIG. 17 is a diagram illustrating a client having an application, filename services, user level I/O drivers, and a layout manager in usermemory address space and having a communication interface in kernelmemory address space according to an embodiment of the system describedherein.

FIG. 18 is a diagram illustrating a client having an application, a filepresentation layer, user level I/O drivers, and a layout manager in usermemory address space and having a VFS and communication interface and akernel memory address space to user memory address space bridge inkernel memory address space according to an embodiment of the systemdescribed herein.

FIG. 19 is a diagram illustrating a client having an application in usermemory address space and having file name services, kernel I/O drivers,a layout manager, and a communication interface in kernel address spaceaccording to an embodiment of the system described herein.

FIG. 20 is a diagram illustrating a client having an application, filename services, user level I/O drivers, and a layout manager in usermemory address space and having a communication interface in kernelmemory address space according to an embodiment of the system describedherein.

FIG. 21 is a diagram illustrating a client having an application, filename services, user level I/O drivers, and a layout manager in usermemory address space and having a communication interface and a kernelmemory address space to user memory address space bridge in kernelmemory address space according to an embodiment of the system describedherein.

FIG. 22 is a diagram illustrating a client having an application in usermemory address space and having a Web Services module, kernel I/Odrivers, a layout manager, and a communication interface in kernelmemory address space according to an embodiment of the system describedherein.

FIG. 23 is a diagram illustrating a client having an application, a WebServices layer, user level I/O drivers, and a layout manager in usermemory address space and having a communication interface in kernelmemory address space according to an embodiment of the system describedherein.

FIG. 24 is a diagram illustrating a client having an application, a WebServices layer, user level I/O drivers, and a layout manager in usermemory address space and having a communication interface and a kernelmemory address space to user memory address space bridge in kernelmemory address space according to an embodiment of the system describedherein.

FIG. 25 is a diagram illustrating a client having a plurality ofapplications, a Web Services layer, file name services, user level I/Odrivers, and a layout manager in user memory address space and having aVFS, a communication interface and a kernel memory address space to usermemory address space bridge in kernel memory address space according toan embodiment of the system described herein.

FIG. 26 is a diagram illustrating different types of servers and a usermanagement interface according to an embodiment of the system describedherein.

FIG. 27 is a flow chart illustrating steps performed in connection withusing security managers servers to obtain credentials for using policymanager servers according to an embodiment of the system describedherein.

FIG. 28 is a diagram illustrating a resource manager table according toan embodiment of the system described herein.

FIG. 29 is a flow chart illustrating steps performed in connection withprocessing resource information to update a resource table according toan embodiment of the system described herein.

FIG. 30 is a flow chart illustrating steps performed in connection withfinding a resource with a desired capability according to an embodimentof the system described herein.

FIG. 31 is a flow chart illustrating steps performed in connection witha metadata server servicing a lease request according to an embodimentof the system described herein.

FIG. 32 is a flow chart illustrating steps performed in connection withmonitoring and recalling expired leases according to an embodiment ofthe system described herein.

FIG. 33 is a flow chart illustrating steps performed in connection withlease recall processing according to an embodiment of the systemdescribed herein.

FIG. 34 is a table illustrating data maintained for files handled by ametadata server according to an embodiment of the system describedherein.

FIG. 35 illustrates a linked list for file leases handled that are by ametadata server according to an embodiment of the system describedherein.

FIG. 36 is a table illustrating data maintained by a metadata server tocorrelate object identifiers and data storage locations according to anembodiment of the system described herein.

FIG. 37 illustrates a table at a metadata location server thatcorrelates object identifier ranges and metadata servers according to anembodiment of the system described herein.

FIG. 38 is a flow chart illustrating processing performed by a metadatalocation server in connection with returning a particular metadataserver according to an embodiment of the system described herein.

FIG. 39 is a flow chart illustrating processing performed by a metadataserver in connection with deleting a file according to an embodiment ofthe system described herein.

FIG. 40 is a flow chart illustrating processing performed by a metadataserver in connection with creating a file according to an embodiment ofthe system described herein.

FIG. 41 is a flow chart illustrating processing performed by a metadataserver in connection with responding to a failed data write operationaccording to an embodiment of the system described herein.

FIG. 42 is a flow chart illustrating processing performed by a metadataserver in connection with responding to a stale mirror update messageaccording to an embodiment of the system described herein.

FIG. 43 is a flow chart illustrating processing performed by a metadataserver in connection with adding an asynchronous copy job to a queueaccording to an embodiment of the system described herein.

FIG. 44 is a flow chart illustrating processing performed by a metadataserver in connection with servicing an asynchronous copy job accordingto an embodiment of the system described herein.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS

Referring to FIG. 1, a diagram illustrates servers 102 coupled to aplurality of clients 104-106. Each of the clients 104-106 represents oneor more processing devices that receives file services from the servers102. Each of the clients 104-106 may or may not be independent of otherones of the clients 104-106. One or more of the clients 104-106 may be amultiprocessing/multiuser system and possibly have multiple independentusers. The clients 104-106 are meant to represent any number of clients.

The file services provided by the servers 102 may include data storageand retrieval as well as related operations, such as data mirroring,cloning, etc. The servers 102 may be implemented using a plurality ofservices (and/or interconnected file servers including SAN components)that are provided by interconnected processing and/or storage devices.In an embodiment herein, each of the clients 104-106 may be coupled tothe servers 102 using the Web, possibly in conjunction with local TCP/IPconnections. However, it is possible for one or more of the clients104-106 to be coupled to the servers 102 using any other appropriatecommunication mechanism and/or combinations thereof to provide thefunctionality described herein.

Referring to FIG. 2A, the client 104 is shown as being coupled to theservers 102 and to one or more other network(s). The other network(s)may include a local area network (LAN). Thus, the client 104 may be agateway between the servers 102 and a LAN to which one or more otherdevices (not shown) may also be coupled. The client 104 may act as alocal file server to the one or more other devices coupled to the LAN byproviding data from the servers 102 to the one or more other devices. Ofcourse, it is possible for one or more other clients to simultaneous actas gateways to the same or different other network(s). Generally, forthe discussion herein, reference to a particular one of the clients104-106 may be understood to include reference to any or all of theclients 104-106 coupled to the servers 102 unless otherwise indicated.

Referring to FIG. 2B, a diagram shows the client 104 being coupled tothe servers 102 and one or more other network(s) (e.g., a LAN) in aconfiguration that is different from that shown in FIG. 2A. In theconfiguration of FIG. 2B, a router 108 is coupled between the servers102 and the client 104. The router 108 may be any conventional routerthat may be accessed by the client 104. In the configuration of FIG. 2B,the client 104 uses only a single connection point to both the servers102 and to the other network(s). In the configuration of FIG. 2B, theclient 104 may act as local file server and gateway between the servers102 and one or more other devices (not shown) coupled to the othernetwork(s). Of course, any other appropriate connection configurationsmay be used by any of the client 104-106 coupled to the servers 102and/or to other network(s).

Referring to FIG. 3, the client 104 is shown in more detail havingserver operations software 122, client software 124, and an interfacelayer 125 that includes a plurality of interfaces 126-128 between theserver operations software 122 and the client software 124. The serveroperations software 122 facilitates the exchange of information/databetween the client 104 and the servers 102 to provide the functionalitydescribed herein. The server operations software 122 is described inmore detail elsewhere herein. The client software 124 represents anysoftware that may be run on the client 104, including applicationsoftware, operating system software, Web server software, etc., that isnot part of the server operations software 122 or the interface layer125. As described in more detail elsewhere herein, it is possible tohave the client software 124 interact with the servers 102 throughdifferent ones of the interfaces 126-128 at the same time.

The file services described herein may be implemented by the servers 102using a set of file objects where a file that is accessed by the clientsoftware includes a metadata file object which points to one or moredata file objects that contain the data for the file.

Accessing the file would involve first accessing the metadata fileobject to locate the corresponding data file objects for the file. Doingthis is described in more detail elsewhere herein. Note, however, thatany appropriate file object mechanism may be used for the systemdescribed herein.

Referring to FIG. 4, a file 130 is shown as including a metadata fileobject 132 and a plurality of data file objects. The metadata fileobject 132 contains information that points to each of the data fileobjects 134-136. Accessing the file includes first accessing themetadata file object 132 and then using information therein to locatethe appropriate one or more of the corresponding data file object134-136.

Referring to FIG. 5, the metadata file object 132 is shown in moredetail as including an object attributes section 142 and a LayoutStorage Object (LSO) tree section 144. The object attributes sectioncontains conventional file-type attributes such as owner id, group id,access control list, last modification time, last access time, lastchange time, creation time, file size, and link count. Many of theattributes are self-explanatory. The last modification time correspondsto the last time that the data for the data objects 134-136 had beenmodified while the last change time corresponds to when the objectmetadata had last been changed. The link count indicates the number ofother objects that reference a particular file (e.g., aliases that pointto the same file). In an embodiment herein, a file and its relatedobjects are deleted when the link count is decremented to zero.

The LSO tree section 144 includes a data structure that includes one ormore maps for mapping the logical space of the file to particular datafile objects. The LSO tree section 144 may also indicate any mirrors forthe data and whether the mirrors are synchronous or asynchronous. LSOtrees and mirrors are described in more detail elsewhere herein.

Referring to FIG. 6, a simple LSO tree 160 is shown as including an LSOroot node 162 and a single map 164. The LSO root node 162 is used toidentify the LSO tree 160 and includes links to one or more map(s) usedin connection with the file corresponding to the LSO tree 160. The map164 maps logical locations within the file to actual data storagelocation. A process that accesses logical storage space of a filerepresented by the LSO tree 160 first uses the LSO root node 162 to findthe map 164 and then uses the map 164 to translate logical addresseswithin the file to an actual data storage locations.

Referring to FIG. 7, an LSO tree 170 is shown as including an LSO rootnode 172 and a plurality of maps 174-176. Each of the maps 174-176 mayrepresent a different range of logical offsets within the filecorresponding to the LSO tree 170. For example, the map 174 maycorrespond to a first range of logical offsets in the file. The map 174may map logical locations in the first range to a first actual storagedevice. The map 175 may correspond to a second range of logical offsetsin the file, different than the first range, which may be mapped to adifferent actual storage device or may be mapped to the same actualstorage device as the map 174. Similarly, the map 176 may correspond toa third range of logical offsets in the file, different than the firstrange and the second range, which may be mapped to a different actualstorage device or may be mapped to the same actual storage device as themap 174 and/or the map 175.

Referring to FIG. 8, an LSO tree 180 is shown as including an LSO rootnode 181 and a pair of replication nodes 182 a, 182 b, which indicatethat the underlying data is to be mirrored (replicated) and whichindicate whether the mirror is synchronous or asynchronous. Synchronousand asynchronous mirrors are discussed in more detail elsewhere herein.The node 182 a has a plurality of children maps 183-185 associatedtherewith while the node 182 b has a plurality of children maps 186-188associated therewith. The replication nodes 182 a, 182 b indicate thatthe data corresponding to the maps 183-185 is a mirror of datacorresponding to the maps 186-188. In some embodiments, the nodes 182 a,182 b may be implemented using a single node 189 to indicatereplication.

A process accessing a file having the LSO tree 180 would traverse thetree 180 and determine that data is mirrored. As discussed in moredetail elsewhere herein, depending upon the type of mirroring, theprocess accessing the LSO tree 180 would either write the data to thechildren of both of the nodes 182 a, 182 b or would provide a message toanother process/server (e.g., the servers 102) that would perform theasynchronous mirroring. Mirroring is discussed in more detail elsewhereherein.

For the system described herein, file objects are accessed by one of theclients 104-106 by first requesting, and obtaining, a lease from theservers 102. The lease corresponds to the file objects for theparticular file being accessed and to the type of access. A lease may befor reading, writing, and/or more some other operation (e.g., changingfile attributes). In an embodiment herein, for objects corresponding toany particular file, the servers 102 may issue only one write lease at atime to any of the clients 104-106 but may issue multiple read leasessimultaneously and may issue read lease(s) at the same time as issuing awrite lease. However, in some embodiments it may be possible to obtain alease for a specified logical range of a file for operations only onthat range. Thus, for example, it may be possible for a first client toobtain lease for writing to a first logical range of a file while asecond client may, independently, obtain a lease for writing to a secondand separate logical range of the same file. The two write leases fordifferent logical ranges may overlap in time without violating thegeneral rule that the system never issues overlapping write leases forthe same data.

The lease provided to the clients 104-106 from the servers 102 includessecurity information (security token) that allows the client appropriateaccess to the data. The security token may expire after a certain amountof time. In an embodiment herein, a client accesses data by providing anappropriate security token for the data as well as clientusers/ownership information. Thus, for example, a user wishing to accessdata would first obtain a lease and then would provide the accessrequest to the servers 102 along with the security token and informationidentifying the owner (client) accessing the data. The servers 102 wouldthen determine whether the access requested by the client waspermissible. After the lease expires (the security token expires), theuser requests the lease again. Data security may be implemented usingconventional data security mechanisms.

After obtaining a lease for accessing a file, a client may then cachethe corresponding metadata, including the LSO tree, into local storageof the client. The client may then use and manipulate the local cachedversion of the metadata and may use the metadata to obtain access to thedata. As described in more detail elsewhere herein, a client does notdirectly modify metadata stored by the servers 102 but, instead, sendsupdate messages to the servers 102 to signal that metadata for a filemay need to be modified by the servers 102.

Referring to FIG. 9, a flowchart 200 illustrates steps performed by aclient in connection with requesting a lease for a file (objectsassociated with a file) for performing operations thereon. Processingbegins at a first step 202 where the client requests the lease for thefile. As discussed in more detail elsewhere herein, a client requestinga lease includes specifying the type of access (e.g., read, write,etc.). Following the step 202 is a test step 204 where it is determinedif the request has been granted. If not, then control transfers from thetest step 204 to a step 206 where processing is performed in connectionwith the lease not being granted to the client. The particularprocessing performed at the step 206 may include, for example, providingan error message to the client process requesting access to the filecorresponding to the lease and/or waiting for an amount of time and thenretrying the request. Note that it is possible that a lease for aparticular file is not available at one time is subsequently availableat another time because, for example, the lease is released by anotherclient in between the first request and the second request. In anyevent, any appropriate processing may be performed at the step 206.Following the step 206, processing is complete.

If it is determined at the test step 204 that the least requested at thestep 202 has been granted, then control transfers from the test step 204to a step 208 where the client performs an operation using the file forwhich the lease was granted. Operations performed at the step 208include reading data and/or writing data. Different types of processingthat may be performed at the step 208 are described in more detailelsewhere herein.

Following the step 208 is a test step 212 where it is determined if theoperations performed at the step 208 require an update. In someinstances, a client may obtain a lease and perform operations that donot affect the file or the underlying file objects. For example, aclient may acquire a lease for reading a file and the operationperformed at the step 208 may include the client reading the file. Insuch a case, no update may be necessary since the file and correspondingfile objects (metadata, data objects, etc.) have not changed. On theother hand, if the client obtains a lease for writing data the file andthe operation performed at the step 208 includes writing data to thefile, then the underlying file objects will have been changed and anupdate message needs to be sent the servers 102. If it is determined atthe test step 212 that an update is necessary, then control passes fromthe test step 212 to a step 214 where an update message is sent by theclient to the servers 102.

Following the step 214, or following the step 212 if no update isnecessary, control passes to a test step 216 where it is determined ifthe client is finished with the file. In some instances, the client mayperform a small number of operations on the file, after which the clientwould be finished with the file at the step 216. In other cases, theclient may be performing a series of operations and may not yet havecompleted all of the operations.

If it is determined at the test step 216 that the client is not finishedwith the file, then control passes from the test step 216 to a test step218 where it is determined if the lease for the file has expired. Notethat a lease may be provided by the servers 102 to the client with aparticular expiration time and/or the associated security token mayexpire. In addition, it may be possible for the servers 102 to recallleases provided to clients under certain circumstances. In either case,the lease may no longer be valid. Accordingly, if it is determined atthe step 218 that the lease has expired (and/or has been recalled by theservers 102), then control passes from the test step 218 back to thestep 202 request the lease again. Otherwise, if the lease has notexpired, then control passes from the test step 218 back to the step 208to perform another iteration.

If it is determined at the test step 216 that the client is finishedwith the file, then control passes from the test step 216 to a step 222where the client releases the lease by sending a message to the servers102 indicating that the client no longer needs the lease. Once theclient releases the lease, it may be available for other clients.Following the step 222, processing is complete.

In an embodiment herein, data file objects may be indicated as havingone of four possible states: current, stale, immutable, or empty. Thecurrent state indicates that the data object is up to date and current.The stale state indicates that the data is not valid but, instead,requires updating, perhaps by some other process. In some instances, thestale state may be used only in connection with mirror copies of data(explained in more detail elsewhere herein). Data may be stale becauseit is a mirror of other data that was recently written but not yetcopied. The immutable state indicates that the corresponding data iswrite protected, perhaps in connection with a previous clone (snapshot)operation. The empty state indicates that no actual storage space hasyet been allocated for the data.

Referring to FIG. 10, a flow chart 240 illustrates steps performed by aclient in connection with performing read operations after obtaining aread lease for a file. Processing begins at a first test step 242 whereit is determined if the data object being read is in the current state.If not, then control transfers from the test step 242 to a step 244where it is determined if the data object being read is in the immutablestate. If it is determined at the step 244 that the data object beingread is in the immutable state or if it is determined at the test step242 that the data object being read is in the current state, thencontrol transfers to a step 246 where the read operation is performed. Aclient reads file data by providing the appropriate data file objectidentifier to the servers 102 as well as providing appropriate securitycredentials. Accordingly, the read operation performed at the step 246includes the client sending an appropriate request to the servers 102and waiting for a result therefrom.

Following the step 246 is a test step 248 where it is determined if theservers 102 have returned a result indicating that the data file objectis unavailable. In some cases, a data file object that is otherwisecurrent or immutable may nevertheless become unavailable. For example,the physical storage space that holds the data file object may becometemporarily disconnected and/or temporarily busy doing some otheroperation. If it is determined at the test step 248 that the data fileobject is available, then control transfers from the test step 248 to atest step 252 where it is determined if the read operation wassuccessful. If so, then control transfers from the test step 252 to astep 254 where the result of the read operation is returned to theprocess at the client that caused the read operation to be performed.The result may include the data that was read and a status indicator.Following the step 254, processing is complete.

If it is determined at the test step 252 that the read operationperformed at the step 246 was not successful, then control transfersfrom the test step 252 to a step 256 where error processing isperformed. The particular error processing performed at the step 256 isimplementation dependent and may include, for example, reporting theerror to a calling process and/or possibly retrying the read operation aspecified number of times. Following the step 256, processing iscomplete.

If it is determined at the test step 244 that the data object being readis not in the immutable state, then control transfers from the test step244 to a test step 258 where it is determined if the data object is inthe stale state. If not, then, by virtue of the test steps 242, 244, 258and process of elimination, the data object is in the empty state. In anembodiment herein, reading a data object in the empty state causes zerosto be returned to the calling process. Accordingly, if it is determinedat the test step 258 that the data object is not in the stale state,then control transfers from the test step 258 to a step 262 where zerosare returned in response to the read operation. Following the step 262,processing is complete.

If it is determined at the test step 258 that the data file object is inthe stale state, or if it is determined at the test step 248 that thedata file object is not available, then control transfers to a test step264 to determine if an alternative version of the data file object isavailable for reading. As discussed in more detail elsewhere herein,there may be multiple versions of the same data file objects that existat the same time due to mirroring. Accordingly, if the data file objectbeing read is in the stale state or otherwise unavailable, it may bepossible to read a mirror copy of the data file object that may be inthe current state. The test performed at the step 264 is described inmore detail elsewhere herein.

If it is determined at the test step 264 that an alternative version ofthe data file object is available, then control transfers from the teststep 264 to a step 266 where the alternative version of the data fileobject is selected for use. Following the step 266, control transfersback to the test step 242 for another iteration with the alternativedata file object.

If it is determined at the test step 264 that an alternative version ofthe data file object is not available, then control transfers from thetest step 264 to a step 268 where the client process waits. In anembodiment herein, it may be desirable to wait for a data file object tobecome current and/or available. Following the step 268, controltransfers back to the step 242 for another iteration. Note that, insteadof waiting at the step 268, processing may proceed from the step 264 tothe step 256 to perform error processing if there is no alternative datafile object available. In other embodiments, it may be possible toperform the step 268 a certain number of times and then, if the datafile object is still unavailable or in the stale state and there is noalternative data file object, then perform the error processing at thestep 256.

Referring to FIG. 11, a flow chart 280 illustrates steps performed by aclient in connection with performing write operations after obtaining awrite lease for a file. Processing begins at a first test step 282 whereit is determined if the data file object to which the write is beingperformed is in the immutable state. If so, then control transfers fromthe step 282 to a step 284 where new actual storage space is allocatedfor the data file object to avoid overwriting the immutable data.Allocating new storage space for a data object may include providing anappropriate request to the servers 102. Following the step 284, controltransfers back to the step 282 to begin the processing for the writeoperation again.

If it is determined at the step 282 that the data file object to whichthe write is being performed is not in the immutable state, then controltransfers from the step 282 to a step 286 where it is determined if thedata file object to which the write is being performed is in the stalestate. If not, then control transfers from the test step 286 to a teststep 288 where it is determined if the data file object to which thewrite is being performed is in the empty state. If so, then controltransfers from the step 288 to the step 284, discussed above, where newphysical storage space is allocated. Following the step 284, controltransfers back to the step 282 to begin the processing for the writeoperation again.

If it is determined at the step 288 that the data file object to whichthe write is being performed is not in the empty state, then controltransfers from the test step 288 to a step 292 where the write operationis performed. Note that the step 292 is reached if the data file objectto which the write operation is being performed is not in the immutablestate, not in the stale state, and not in the empty state (and thus isin the current state). A client writes file data by providing theappropriate data file object location identifier to the servers 102 aswell as providing appropriate security credentials. Accordingly, thewrite operation performed at the step 292 includes the client sending anappropriate request to the servers 102 and waiting for a resulttherefrom.

Following the step 292 is a test step 294 where it is determined if thewrite operation performed at the step 292 was successful. If so, thencontrol transfers from the test step 294 to a test step 296 where it isdetermined if there are synchronous mirrors of the data file object towhich the write is being performed. The test performed at the step 296may include, for example, determining if a parent node of the data fileobject in the file LSO tree indicates replication. If not, then controltransfers from the test step 296 to a step 298 where an update (message)is sent to the servers 102 indicating that the write had been performed.Following the step 298, processing is complete.

If it is determined at the test step 296 that there are synchronousmirrors of the data file object to which the write is being performed,then control passes from the test step 296 to a step 302 where the datathat was written at the step 292 is also written to the synchronousmirror(s). The processing performed at the step 302 is discussed in moredetail elsewhere herein. Following the step 302, control transfers tothe step 298, discussed above, where an update (message) is sent to theservers 102. Following the step 298, processing is complete.

If it is determined at the test step 294 that the write operationperformed at the step 292 was not successful, or if it is determined atthe test step 286 that the data file object to which the write operationis being performed is in the stale state, then control transfers to astep 304 where the data file object to which the write is attempting tobe performed is removed from the client's local copy of the LSO tree. Atthe end of the write operation illustrated by the flow chart 280, theclient may inform the servers 102 (at the step 298) of the difficulty inwriting to the data object so that the servers 102 can take appropriateaction, if necessary.

Following the step 304 is a test step 306 where it is determined if analternative version of the data is available. As discussed in moredetail elsewhere herein, there may be multiple versions of the same datafile objects that exist at the same time due to mirroring. Accordingly,if the data file object to which the write operation is being performedis stale or otherwise cannot be written to, it may be possible to writeto a mirror copy of the data. The test performed at the step 306 is likethe test performed at the step 264 and is described in more detailelsewhere herein. If it is determined at the test step 306 that analternative version of the data corresponding to the data file object isavailable, then control transfers from the test step 306 to a step 308where the alternative version is selected for writing. Following thestep 308, control transfers back to the test step 282 for anotheriteration with the alternative data file object.

If it is determined at the test step 306 that an alternative version ofthe data corresponding to the data file object is not available, thencontrol transfers from the test step 306 to a step 312 to perform errorprocessing if there is no alternative available. The particular errorprocessing performed at the step 312 is implementation dependent and mayinclude, for example, reporting the error to a calling process and/orpossibly retrying the write operation a specified number of times beforereporting the error. Following the step 312, control transfers to thestep 298, discussed above, to send update information to the servers102. Following the step 298, processing is complete.

Referring to FIG. 12, a flow chart 320 illustrates in more detail stepsperformed in connection with the alternative available test step 264 ofFIG. 10 and/or the alternative available test step 306 of FIG. 11.Processing begins at a first test step 322 where it is determined if thefile has any mirror data file objects at all. In some instances, a filemay not use mirrors, in which case there would be no alternative copyavailable. Accordingly, if it is determined at the test step 322 thatthe file does not have any mirror data file objects, then controltransfers from the test step 322 to a step 324 where a value is returnedindicating that no alternative copies are available. Following the step324, processing is complete.

If it is determined at the test step 322 that mirror copies areavailable, then control transfers from the test step 322 to a step 326where a pointer is made to point to a first mirror data file object. Forthe processing discussed herein, a pointer may be used to iteratethrough mirror data file objects to find a useable data file object.Following the step 326 is a test step 328 where it is determined if thepointer is past the end of the list of mirror data file objects (hasiterated through all of the mirror data file objects). If so, thencontrol passes from the test step 328 to the step 324, discussed above,to return a value that indicates that no alternatives are available.

If it is determined at the test step 328 that the pointer is not pastthe end of a list of mirror data file objects, then control transfersfrom the test step 328 to a test step 332 where it is determined if thepointer points to a data file object in a stale state. If so, thencontrol transfers from the test step 332 to a step 334 where the pointeris made to point to the next data file object to be examined. Followingthe step 334, control transfers back to the step 328, discussed above,for another iteration. If it is determined at the test step 332 that thepointer does not point to a data file object in the stale state, thencontrol transfers from the test step 332 to a step 336 where the datafile object that is pointed to by the pointer is returned as analternative data file object that may be used by the calling process.Following the step 336, processing is complete.

Referring to FIG. 13, a flow chart 350 illustrates in more detailoperations performed in connection with the step 302 of the flow chart280 of FIG. 11 where data that has been written is copied to a number ofsynchronous mirrors (mirror data file objects). Processing begins at afirst step 352 where a pointer that is used to iterate through themirror data file objects is set to point the first one of the mirrordata file objects. Following the step 352 is a test step 354 where it isdetermined if the pointer used for iterating through the mirror datafile objects points past the end (i.e., if all of the mirror data fileobjects have been processed). If so, then processing is complete.Otherwise, control transfers from the test step 354 to a test step 356where it is determined if the status of the mirror data file objectpointed to by the pointer indicates that the mirror data file object iscurrent. If not, then control passes from the test step 356 to a teststep 358 where it is determined if the status of the mirror data fileobject pointed to by the pointer indicates that the mirror data fileobject is in the stale state. If so, then control passes from the teststep 358 to a step 362 where the mirror data file object is removed fromthe client's local copy of the LSO tree. In an embodiment herein, asynchronous mirror data file object should not be in a stale state and,if that occurs, it may indicate an error condition. Accordingly,following the step 362 is a step 364 where information about the stalemirror is sent to the servers 102, which may perform recovery processingin response thereto.

Note that if a mirror data file object is neither in the stale state norin the current state, then the mirror data file object is either in theempty state or in the immutable state. In either case, it may benecessary to allocate new space for a data file object to which the datais to be written. Accordingly, if it is determined at the test step 358that the data file object is not in the stale state, then control passesfrom the test step 358 to a step 366 where new space is allocated forthe mirror data file object. Following the step 366 is a step 368 wherethe data that is being copied across synchronous mirror data fileobjects is written to the mirror data file object pointed to by thepointer used to iterate through the mirror data file objects. Note thatthe step 368 may also be reached from the test step 356 if it isdetermined that the mirror data file object is current. Following thestep 368 is a step 372 where the pointer used to iterate through themirror data file objects is made to point to the next one. Note that thestep 372 is also reached following the step 364. Following the step 372,control transfers back to the test step 354 for another iteration.

The system described herein may access file objects using objectidentifiers. In an embodiment herein, each file object that is storedamong the servers 102 may be assigned a unique object identifier thatidentifies each file object and distinguishes each file object fromother file objects in the system. However, many applications use a filenaming structure and/or a hierarchical directory to access files anddata therein. For example, a file name “C:\ABC\DEF\GHI.doc” indicates afile called “GHI.doc” stored in a sub-directory “DEF” that is stored inanother directory “ABC” located on a root volume “C”. A nested directorystructure may be provided by implementing directories as special filesthat are stored in other directories. In the example given above, thesub-directory “DEF” may be implemented as a file stored in the directory“ABC”.

The system described herein may present to applications a conventionalnaming structure and directory hierarchy by translating conventionalfile names into file object identifiers. Such a translation service maybe used by other services in connection with file operations. In anembodiment herein, each directory may include a table that correlatesfile names and sub-directory names with file object identifiers. Thesystem may examine one directory at a time and traverse sub-directoriesuntil a target file is reached.

Referring to FIG. 14, a flow chart 380 illustrates steps performed inconnection with providing a file name translation service (file nameservice) that translates a conventional hierarchical file name into afile object identifier. The file name service may receive a conventionalhierarchical file name as an input and may return an object identifier(or, in some cases, an error). Processing begins at a first step 382where the file name service receives a file name, such as a conventionalhierarchical file name. Following the step 382 is a test step 384 whereit is determined if the syntax of the file name is OK. Checking thesyntax of a hierarchical file name is know and includes, for example,checking that only appropriate characters have been used. If it isdetermined at the test step 384 that the syntax is not OK, then controltransfers from the test step 384 to a step 386 where an error indicator(error message) is returned to the calling process. Following the step386, processing is complete.

If it is determined at the test step 384 that the syntax of the providedname is OK, then control transfers from the test step 384 to a step 388where the root directory is read. In an embodiment herein, all file namepaths begin at a single common root directory used for all file objectsstored in the servers 102. In other embodiments, there may be multipleroot directories where specification of a particular root directory maybe provided by any appropriate means, such as using a volume identifier,specifically selecting a particular root directory, etc.

Following the step 388 is a test step 392 where it is determined if thetarget file (or sub-directory that is part of the file name path) is inthe directory that has been read. If not, then control passes from thetest step 392 to the step 386, discussed above, where an error isreturned. In some embodiments, the file-not-found error that resultsfrom the test at the step 392 may be different from the syntax errorthat results from the test at the step 384.

If it is determined that the target file or a sub-directory that is partof the file name path is in the directory that has just been read, thencontrol passes from the test step 392 to a test step 394 where it isdetermined if the directory that has just been read contains the targetfile (as opposed to containing a sub-directory that is part of the filename path). If so, then control passes from the test step 394 to a step396 where the object identifier of the target file object is returned tothe calling process. Following the step 396, processing is complete.

If it is determined at the test step 394 that the directory that hasjust been read contains a sub-directory that is part of the file namepath, then control transfers from the test step 394 to a step 398 wherethe sub-directory is read so that the sub-directory becomes thedirectory being examined. In effect, processing at the step 398traverses the chain of subdirectories to eventually get to the targetfile. Following the step 398, control transfers back to the step 392,discussed above, for a next iteration.

Referring to FIG. 15, a diagram shows the client 104 as including useraddress memory space and kernel address memory space. In an embodimentherein, user address memory space is memory space that is generally usedby user applications and related processes while kernel address memoryspace is memory space that is generally accessible only by systemprocesses, such as an operating system kernel and related processes. Asdiscussed in more detail herein, it is possible to have differentportions of the system described herein reside and operate in the usermemory space and/or the kernel memory space. In addition, it is possiblefor the client 104 to have multiple different interfaces to access fileobjects at the servers.

In FIG. 15, the client 104 is shown as including an application in theuser memory address space and a virtual file system (VFS), file nameservices, kernel I/O drivers, a layout manager, and a communicationinterface in the kernel memory address space. The VFS is an abstractionlayer on top of a more concrete file system. The purpose of a VFS is toallow client applications to access different types of concrete filesystems in a uniform way. The VFS allows the application running on theclient 104 to access file objects on the servers 102 without theapplication needing to understand the details of the underlying filesystem. The VFS may be implemented in a conventional fashion bytranslating file system calls by the application into file objectmanipulations and vice versa. For example, the VFS may translate filesystem calls such as open, read, write, close, etc. into file objectcalls such as create object, delete object, etc.

The VFS may use the file name services, described elsewhere herein, totranslate file names into object identifiers. The kernel I/O driversprovide an interface to low-level object level I/O operations. Thekernel I/O drivers may be modeled after, and be similar to, Linux I/Odrivers. The layout manager may perform some of the processing on LSOtrees corresponding to files, as discussed in more detail elsewhereherein. The communication interface provides communication between theclient 104 and the servers 102. The communication interface may beimplemented using any appropriate communication mechanism. For example,if the client 104 communicates with the servers 102 via an Internetconnection, then the communication interface may use TCP/IP tofacilitate communication between the servers 102 and the client 104.

The application of FIG. 15 may correspond to the client software 124 ofFIG. 3. The VFS of FIG. 15 may correspond to one of the interfaces126-128 of FIG. 3. The file name services, kernel I/O drivers, layoutmanager, and communication interface of FIG. 15 may correspond to theserver operations software 122 of FIG. 3. Similar correlation betweencomponents of FIG. 3 and other figures may also be found.

Referring to FIG. 16, a flow chart 410 illustrates steps performed by aVFS to provide file services in connection with an application runningon the client 104. Processing begins at a first step 412 where a filesystem operation requested by an application may be translated into oneor more object operations. For example, a file operation to open a filefor reading may be converted to object operations that include obtainingan object lease for reading as discussed elsewhere herein. Following thestep 412 is a step 414 where the VFS translates the file name into anobject identifiers using the file name services discussed above inconnection with FIG. 14. Operations that follow may be performed usingthe object identifiers obtained at the step 414.

Following the step 414 is a test step 416 where it is determined if therequested operation requires the LSO tree. As discussed elsewhereherein, operations such as read, write, etc. use LSO trees correspondingto file objects. However, some possible file operations may not requireaccessing a corresponding LSO tree. If it is determined at the test step416 that the LSO tree is needed, then control transfers from the teststep 416 to a step 418 where the VFS accesses the LSO manager to performthe necessary operations. For example, for a read operation, the LSOmanager may perform processing like that illustrated in the flow chart240 of FIG. 10. Following the step 418, or following the step 416 if theLSO is not needed, is a step 422 where the operations are passed to lowlevel kernel I/O drivers (e.g., via one or more appropriate API's). Thekernel I/O drivers use the communication module to communicate betweenthe client 104 and the servers 102 in connection with performing therequested operation(s). In instances where the application running onthe client 104 has requested data and/or other information from theservers 102, the data and/or information may be passed back up throughthe communication interface, kernel I/O drivers, etc. to the VFS andultimately to the application.

Referring to FIG. 17, the client 104 is shown as having an application,file name services, user level I/O drivers, and a layout manager allprovided in user memory address space. The functionality of the VFS thatwas shown in FIG. 15 and described above may be performed instead bylibrary routines linked to the application, and thus are part of theapplication. These routines would provide functionality like thatdiscussed above in connection with FIG. 16. Accordingly, it is theapplication that uses the file name services and makes calls to the userlevel I/O drivers (like the kernel I/O drivers) and to the layoutmanager. The communication interface is still maintained in the kernelmemory address space.

Note that, for the configuration of FIG. 15, modifications are providedby modifying system processes (the operating system), which isdisadvantageous for a number of reasons. For example, if the client 104is a multiuser computing system, then modifying the operating system mayinvolve restarting the entire system and thus disrupting all of theusers. In contrast, the configuration of FIG. 17 is advantageous sinceit allows modification of the system in the application/user memoryaddress space so that the operating system of the client 104 does notneed to be modified. However, the configuration of FIG. 17 does not usea VFS, and thus does not obtain the advantageous separation of theapplication from the file system that is provided by the VFS in FIG. 15.

Referring to FIG. 18, the client 104 is shown as having an applicationin user memory address space that accesses file objects through a VFS inkernel memory address space like that illustrated in FIG. 15. However,the file name services, I/O drivers, and the layout manager all residein the user memory address space like the system illustrated in FIG. 17.The VFS communicates with components in the user memory address spacethrough a bridge between kernel memory address space and user memoryaddress space, such as a FUSE (or similar) interface. The bridge allowsfile system components to be provided in user memory space instead ofkernel address memory space while still preserving the VFS in the kerneladdress memory space. Thus, the configuration illustrated by FIG. 18provides the advantages of using a VFS, as illustrated in theconfiguration of FIG. 15, along with the advantages of having filesystem components in the user address memory space, as illustrated inthe configuration of FIG. 17.

It is possible in some instances to have applications and/or otherprocessing in the user memory address space of the client 104 accessfile objects directly, rather than through a file services layer likethe VFS and/or equivalent functionality provided by user linkablelibraries (e.g., the configuration illustrated in FIG. 17). Accessingfile objects directly may include invoking routines that create objects,read objects, modify objects, delete objects, etc. In such a case, theapplication would need to know how to interpret and/or manipulate theobject data, which may not always be desirable. For example, anapplication that accesses file objects through the VFS may not need totake into account (or even know about) the structure of an LSO treewhile an application that accesses objects directly may need to use theLSO tree. On the other hand, removing the file services layer mayprovide an opportunity for optimizations not otherwise available. Notethat, since the servers 102 exchange object information/operations withthe clients 104-106, the servers 102 may not need to distinguish or beable to distinguish between application on the clients 104-106 using afile system interface (file services like the VFS) and those that arenot.

Referring to FIG. 19, the client 104 is shown as including anapplication in the user memory address space and kernel I/O drivers, alayout manager, and file name services in the kernel memory addressspace. The configuration illustrated in FIG. 19 is like that illustratedin FIG. 15, except that the VFS is not used. In the configurationillustrated in FIG. 19, the application could directly access the filename services, the kernel I/O drivers, and the layout manager. Thecommunication interface in the kernel memory address space communicateswith the servers 102 just as in other configurations. The direct accessillustrated in FIG. 19 allows applications to manipulate file objects(via, for example, appropriate API's) while access via the VFS (orsimilar) allows applications to accesses file objects indirectly throughfile system calls to the VFS.

Referring to FIG. 20, the client 104 is shown as having an application,user level I/O drivers, a layout manager, and file name services allprovided in user memory address space. The configuration shown in FIG.20 is like that shown in FIG. 17. However, as set forth above, theconfiguration of FIG. 17 includes file service libraries that are linkedinto, and thus part of, the application. In contrast, in theconfiguration of FIG. 20, the application is not linked into librarieswith extensive file services. Instead, like the application of theconfiguration illustrated in FIG. 19, the application in theconfiguration of FIG. 20 uses minimal file services and, instead, usesand operates upon file objects directly using the user level I/Odrivers, the layout manager and, if a file name translation is needed,the file name services.

Referring to FIG. 21, the client 104 is shown as having an applicationin user memory address space and a bridge in the kernel memory addressspace. File name services, user level I/O drivers, and a layout managerare provided in user memory address space. However, unlike theconfiguration of FIG. 20, the application does not make direct calls tothe file system components in the user memory address space. Instead,the application calls the file system components indirectly through thebridge. Just as with the configuration illustrated in FIG. 18, theconfiguration of FIG. 21 advantageously locates file system componentsin the user memory address space and, at the same time, provides akernel memory address space layer between the application and the filesystem components.

Referring to FIG. 22, the client 104 is shown as having an applicationin user memory address space and a Web Services module in kernel memoryaddress space. The application may be a Web server application or anyapplication that handles communication with the Web. In an embodimentherein, the application allows communication with the client 104, whichacts as a Web server to other computing devices (not shown) that accessthe client 104 through a Web connection.

The configuration illustrated in FIG. 22 provides Web Services in amanner similar to the file services and/or file object access providedby other configurations. However, the Web Services receivesrequests/data via a Web data protocol, such as HTML, and providesresponses/data also in a Web data protocol, which may be the same ordifferent from the protocol used for requests/data. Operations handledby the Web Services may include object-level operations such as createobject, delete object, read object, modify object, modify objectmetadata, etc. It is also possible to provide more file system leveloperations, via the Web Services, that open files, read data from files,etc. by including at least some of the functionality of the fileservices, described elsewhere herein, with the Web Services. The WebServices may present to the other computing devices a conventionalwell-known Web Services protocol, such as REST or SOAP, or may provideany other appropriate protocol.

Referring to FIG. 23, the client 104 is shown as having an application,Web Services, user level I/O drivers, and a layout manager in usermemory address space. The application may include a Web connection thatallows communication with the client 104, which acts as a Web server toother computing devices (not shown) that access the client 104 throughthe Web connection. The configuration of FIG. 23 is like that of FIG. 17and FIG. 20. The advantages of the configuration shown in FIG. 23 overthe configuration shown in FIG. 22 is that, generally, changes to theconfiguration shown in FIG. 23 do not require reconfiguring kernelmemory address space processes.

Referring to FIG. 24, the 104 is shown as having an application, WebServices, user level I/O drivers, and a layout manager in user memoryaddress space. The application may include a Web connection that allowscommunication with the client 104, which acts as a Web server to othercomputing devices (not shown) that access the client 104 through the Webconnection. A bridge is provided in the kernel memory address space. Theconfiguration of FIG. 24 has similar advantages to the configurationshown in FIG. 23, but also has the advantages provided by providing thebridge, discussed elsewhere herein.

Referring to FIG. 25, the client 104 is shown as having a plurality ofapplications in user memory address space, each of which may use adifferent interface to access file objects of the servers 102. Each ofthe applications shown in FIG. 25 is meant to represent one or moreapplications. Accordingly, APP1 may present one or more applicationsthat access file objects at the servers 102 using a Web Servicesinterface. The APP1 application may include a Web connection that allowscommunication with the client 104, which acts as a Web server to othercomputing devices (not shown) that access the client 104 through the Webconnection. APP2 may represent one or more applications that access fileobjects at the servers 102 using the VFS, and APP3 may represent one ormore applications that directly operate on file objects at the servers102. The different interfaces may operate at the client 104 at the sametime.

Note that may other combinations of configurations, includingillustrated configurations, are possible so that the client 104 maysimultaneously present to applications thereon different interfaces. Forexample, it is possible to combine the configurations illustrated inFIGS. 15, 19, and 22 and/or combine the configurations of FIGS. 17, 20,and 23. Other combinations, including combinations of only twoillustrated configurations, are also possible. The servers 102 providethe file objects to the clients 104 provided: 1) the requesting clienthas appropriate authorization for whatever operation is requested forthe file objects; and 2) there is no conflict with any previous request.For example, in systems where only one client is allowed to write to anobject at any one time, the servers 102 would not allow one of theclients 104-106 to modify a particular object while another one of theclients 104-106 is also modifying the object.

Referring to FIG. 26, the servers 102 are shown in more detail asincluding one or more policy manager servers 402, one or more securitymanager servers 403, one or more audit servers 404, one or more metadataservers 405, one or more resource manager servers 406, one or more datastorage servers 407, and one or more metadata location servers 408. Eachof the servers 402-408 may be implemented as one or more unitaryprocessing devices capable of providing the functionality describedherein. For the discussion herein, reference to servers should beunderstood as a reference to one or more servers. The servers 402-408may be interconnected using any appropriate data communicationmechanism, such as TCP/IP, and may be coupled to the clients 104-106(not shown in FIG. 26) using any appropriate data communicationmechanism, such as TCP/IP.

The servers 102 may include a user management interface 412 thatfacilitates system management. The user management interface 412exchanges data with the policy management servers 402, the securitymanagement servers 403, and the audit servers 404 to affect how theservers 102 interact with the clients 104-106 and corresponding users.

The data may be provided through the user management interface 412 inany one of a number of ways, including conventional interactive computerscreen input and data file input (e.g., a text file having usermanagement commands). The data may include information that correlatesclasses of users and storage parameters such as Quality of Service(QOS), RAID protection level, number and geographic location(s) ofmirrors, etc. For example, an administrator may specify through the usermanagement interface 412 that users of a particular class (usersbelonging to a particular group) store data file objects on storagedevices having a particular RAID level protection.

The servers 102 also include physical storage 414 coupled to the datastorage servers 407. Although the physical storage 414 is shown as asingle item in FIG. 26, there may be any number of separate physicalstorage units that may be geographically dispersed. In addition, theremay be different types of physical storage units having differentcapabilities. Accordingly, the physical storage 414 genericallyrepresents one or more instances of physical data storage for the systemthat is managed by the data storage servers 407, as explained in moredetail below.

Data modifications, including modifications of metadata file objectsand/or policies that affect handling/creation of metadata file objects,require appropriate security credentials. Accordingly, the securitymanager servers 403 may restrict/inhibit the ability of certainadministrators (users) to modify and/or create policies for classes ofusers.

Referring to FIG. 27, a flow chart 430 illustrates steps performed bythe user management interface 412 to obtain and use security credentialsfor accessing the policy manager servers 402. Processing begins at afirst step 432 where the user management interface 412 sends a requestto the security manager servers 403 to obtain a token (or otherappropriate security credentials) for the operation to be performed bythe user management interface 412. Following the step 432 is a test step434 where it is determined if the token has been granted (provided). Insome instances, the security manager servers 403 may not issue asecurity token at all. For example, if the administrator (user) does nothave sufficient rights to perform the desired function.

If the security token is not granted, then control passes from the step434 to a step 436 where processing is performed in connection with thesecurity token not being granted. The operations performed at the step436 may including providing a message to the administrator (user)through the security management interface 412 indicating that theadministrator does not have sufficient rights to perform the desiredoperation. Following the step 436, processing is complete.

If it is determined at the test step 434 that a security token has beengranted (provided) by the security manager servers 403, then controlpasses from the test step 434 to a step 438 where the user managementinterface 412 provides the security token, and user id information, tothe policy manager servers 402. Of course, information indicating thedesired operation/modification may also be provided at the step 438.Following the step 438 is a test step 442 where it is determined if thepolicy manager servers 402 have allowed the requested operation. Notethat, in some instances, the policy manager servers 402 may not allow aparticular operation even though the security manager servers 403 haveprovided a security token. For example, if the user id and the userindicated by the security token do not match and/or if the requestedoperation and the operation indicated by the security token do notmatch.

If it is determined at the test step 442 that the requested operation isnot allowed, then control passes from the test step 442 to the step 436,described above, where processing is performed to indicate that thereare security issues. The processing performed at the step 436 mayinclude providing a message to an administrator (user) indicating thatthe operation cannot be performed because of insufficient securityrights. The message provided when the step 436 is reached from the step442 may be different than the message provided when the step 436 isreached from the step 434.

If it is determined at the test step 442 that the requested operation isallowed, then control passes from the test step 442 to a step 444 wherethe operation is performed. Performing the operation at the step 444 mayinclude modifying policy data, as described in more detail elsewhereherein. Following the step 444, processing is complete.

Thus, an administrator (user) accessing the policy manager servers 402would first provide identification information to the security managerservers 403 that would return a security token (perhaps having anexpiration time). The administrator presents the token andidentification information to the policy manager servers 402, whichwould decide to grant or deny access based on the token and theidentification information. Note that the security mechanism illustratedby the flow chart 430 of FIG. 27 may be extended to be used inconnection with accessing any of the servers 402-408 and/or other data.For example, one of the clients 104-106 could obtain/modify file objectsby first requesting a security token from the security manager servers403 prior to performing an operation that includes operations with fileobjects. Accordingly, for the discussion herein, it can be assumed thataccess to file objects, servers, etc. includes appropriate securityprocedures like those illustrated in FIG. 27.

The policy manager servers 402 handle placement and protection of fileobjects. An administrator (user) may input, through the user managementinterface 412, different policy templates that may be assigned todifferent ones of the clients 104-106, different users, differentclasses of users, or any other appropriate group. For example, a policytemplate may indicate that, for a particular group of users, whenever anew file is created, a mirror will be created that is geographicallyfarther from the initial data set by at least a certain distance. Insuch a case, when a first user of the group creates an initial data setin New York, a mirror may be automatically created in Los Angeles while,when a second user creates an initial data set in Los Angeles, a mirrormay be created in New York.

The audit servers 404 may be used to provide system auditing capability.A user may communicate to the audit servers 404 through the usermanagement interface 412. The user may indicate the type of informationto be audited (tracked).

The resource manager servers 406 keep track of available systemresources. In some instances, the resource manager servers 406 mayinteract with the policy manager servers 402 in connection withestablishing policy templates and/or assigning policy templates. In somecases, a user may attempt to construct a policy template that isimpossible to fulfill if assigned to a group. For example, if all of thephysical data storage is in a single geographic location, then it wouldnot be appropriate to have a policy template indicating that new filesshould include a mirror that is geographically distant from the initialdata set.

The resource manager servers 406 receive information from othercomponents of the system in order to be able to keep track whichresources are available. Whenever a resource is added to the system, theresource or another component reports that information to the resourcemanager servers 406. For example, if new physical storage is added tothe system, the new physical storage itself, or a corresponding one ofthe data storage servers 407, sends a message to the resource managerservers 406. Similarly, if a resource becomes full (e.g., a physicaldisk is full) or is removed from the system (planned removal orunplanned resource failure), information is provided to the resourcemanager servers 406. In an embodiment herein, system resources maycorrespond to portions of the physical storage 414 and/or data servers407 that manage the physical storage 414.

Referring to FIG. 28, a resource table 460 is shown as including aplurality of entries 462-464, each of which corresponds to a particularstorage resource. Although only three entries are shown, the table 460may contain any number of entries. The table 460 may be implementedusing any appropriate technique, including an array, linked list, etc.

Each of the entries 462-464 includes a resource field identifying aparticular resource corresponding to the entry. In an embodiment herein,each of the entries 462-464 may correspond to a particular one of thedata storage servers 407 and/or a portion thereof. Each of the entries462-464 includes a status field corresponding to the status of thecorresponding resource. In an embodiment herein, the status field mayindicate that a resource is on-line (available) or off-line(unavailable). The status field may also indicate the percentage of usedspace of a resource, and perhaps indicate any performance degradation.

Each of the entries 462-464 may also include a capabilities field thatindicates the capabilities of the corresponding resource. In anembodiment herein, when the resources represent storage areas, thecapabilities field may indicate particular capabilities of acorresponding storage area. Particular capabilities may include theresource being green (low energy use through, for example, spinningdisks down when not in use), capable of data deduplication (maintainingonly a single copy of data that is otherwise duplicated), capable ofvarious RAID configurations, etc. The capabilities field may indicateany appropriate data storage capabilities.

Referring to FIG. 29, a flow chart 480 indicates operation of theresource manager servers 406 in connection with maintaining informationabout system resources. Processing begins at a first step 482 where theresource manager servers 406 are initialized with information aboutresources. The initialization processing performed at the step 482 maytake any form, including loading a fixed table of initially availableresources, having the resource manager servers 406 poll systemresources, etc.

Following the step 482 is a test step 484 where the resource managerservers 406 wait for new information to be provided. In an embodimentherein, after initialization, the resource manager servers 406 wait toreceive information from other system components. In other embodiments,it may be possible to have the resource manager servers 406 periodicallypoll system components to see if anything has changed. If it isdetermined at the test step 484 that no new information is available,control loops back on the test step 484 to continue polling.

Once it is determined at the test step 484 that new information isavailable, then control transfers from the test step 484 to a test step486 where it is determined if the new information relates to a newresource added to the system. If so, then control transfers from thetest step 486 to a step 488 where the new entry is added to the resourcetable that is managed by the resource manager servers 406. Following thestep 488, control transfers back to the step 484 to continue waiting fornew information.

If it is determined at the step 486 that the received resourceinformation does not related to a new resource (and thus relates to achange of an existing resource), then control transfers from the step486 to a step 492 where the existing entry is located in the resourcetable. Following the step 492 is a test step 494 where it is determinedif the capability is being changed for the modified resource. Thecapability of a resource may change under many different circumstances.For example, a resource may degrade and lose capabilities, a resourcemay be modified/enhanced and gain capabilities, a local manager of aresource may decide to make certain capabilities available/unavailable,etc.

If it is determined at the step 494 that the capabilities of a resourcehave changed, then control transfers from the test step 494 to a step496 to change the capabilities field for the resource being modified.Otherwise, control transfers from the test step 494 to a step 498 tochange the status field of the resource being modified (e.g., resourceis full, resource is off-line, resource is on-line, etc.). Followingeither the step 496 or the step 498, control transfer back to the step484, discussed above, for another iteration.

Note that the resource manager servers 406 may represent a plurality ofseparate computing devices that may be dispersed throughout the system.Furthermore, each of the separate computing devices may maintain its owncopy of the resource table. The separate computing devices that are usedto implement the resource manager servers 406 may or may not shareresource information and may or may not receive the same resource statusmessages. In instances where information sharing and/or receipt ofstatus messages is not perfect, then each of the computing devices mayhave a somewhat different version of the resource table and it ispossible for no one version of the resource table to reflect acompletely accurate picture of the exact state of all of the resourcesof the system.

The physical storage 414 may be provided using relatively inexpensiveoff-the-shelf mass produced storage hardware. In an embodiment herein,at least some of the physical storage 414 may be implemented usingserial ATA disk drives, which are available from a number ofmanufactures such as Seagate and Western Digital. As discussed elsewhereherein, the physical storage may be geographically dispersed. However,each portion of the physical storage may be managed/controlled by atleast one of the data storage servers 407, which may be implementedusing conventional computing devices local to the corresponding portionof the physical storage 414.

In an embodiment herein, the data storage servers 407 may present an OSDStandard interface to the system. Thus, the servers 102 and/or theclients 104-106 may access physical storage 414 through the data storageservers 407 using OSD calls and may receive information/data accordingto the OSD protocol. In addition, the data storage servers 407 mayhandle managing/posting the capabilities and status of differentportions of the physical storage 414. Thus, for example, when a portionof the physical storage 414 is managed by a particular server of thedata storage servers 407, the particular server may send a message tothe resource manager servers 406 indicating the new status.

Referring to FIG. 30, a flow chart 510 illustrates steps performed bythe resource manager servers 406 in connection with servicing an inquiryfor a resource with particular capabilities (i.e., finding a resourcewith particular capabilities). Processing begins at a first step 512where a pointer, used to iterate through each entry of the resourcetable, is set to point to the first entry. Following the step 512 is atest step 514 where it is determined if the pointer points past the endof the table (i.e., all entries have been examined). If so, then controlpasses from the test step 514 to a step 516 where a result indicating nomatch for the requested capabilities is returned by the resource managerservers 406. Following the step 516, processing is complete.

If it is determined at the test step 514 that the pointer used toiterate through the entries does not point past the end of the table,then control transfers from the test step 514 to a test step 518 whereit is determined if the entry currently indicated by the pointer is amatch for the requested capability. Note that the test at the step 518may include checking the status of a resource to ensure that theresource is on-line and not full or otherwise unusable. If it isdetermined at the step 518 that the resource indicated by the pointerhas the requested capability, then control transfers from the test step518 to a step 522 where the resource manager servers 406 return anindicator indicating the matching resource. Following the step 522,processing is complete.

If it is determined at the step 518 that the resource indicated by thepointer does not have the requested capability (or is off-line, full,etc.), then control transfers from the test step 518 to a step 524 wherethe pointer is incremented. Following the step 524, control transfersback to the step 514, discussed above, for another iteration.

The LSO trees that are part of the metadata objects for files arecreated, maintained, and manipulated by the metadata servers 405. Themetadata servers 205 handle updates from the clients 104-106 inconnection with manipulation of file objects (e.g., at the step 214 ofthe flow chart 200 of FIG. 9, at the step 298 of the flow chart 280 ofFIG. 11, etc.). The metadata servers 205 may also handle any actions,besides modifying the LSO tree, that may need to be performed inconnection with the updates. The metadata servers 205 also handle leasesobtained for file objects.

Referring to FIG. 31, a flow chart 550 illustrates steps performed bythe metadata servers 205 in connection with servicing a request for alease for a file. As discussed elsewhere herein, a lease may berequested by one of the clients 104-106. However, it is also possiblefor one of the components of the servers 102 to request a lease. It iseven possible for one of the metadata servers 405 to request a lease.One of the metadata servers 405 may request a lease in connection withfile maintenance (e.g., repairing mirrors), as described in more detailelsewhere herein. In an embodiment herein, leases are maintained by themetadata servers 405 in a table that indicates the entity that hasacquired the lease, the type of lease (e.g., read or write), andpossibly an expiration time.

In addition, as discussed elsewhere herein, it is possible to performlease operations on ranges of logical addresses in a file so that, forexample, one set of lease operations may be performed on logicaladdresses A-B for a file while another set of lease operations may beindependently performed for logical addresses C-D for the same file,where A-B does not overlap C-D. In a system where only one write leaseis issued at a time, it may still be possible for one entity to acquirea write lease for the A-B portion of a file while another independententity simultaneously acquires a write lease for the C-D portion of thesame file. Accordingly, for the discussion herein, in appropriateinstances, a reference to a file or files should be understood toinclude non-overlapping portions of a file or files.

Processing begins at a first test step 552 where it is determined if therequested lease is available. The test at the step 552 determines if therequestor has appropriate security credentials, if the correspondingdata file exists, etc. Also, as discussed in more detail elsewhereherein, leases may be purposely made unavailable in certaincircumstances. If it is determined at the test step 552 that the leaseis not available, then control transfers from the test step 552 to astep 554 where a failure indicator is returned to the requestor. Thefailure indicator may include a reason for the failure (e.g., impropersecurity credentials, file does not exist, etc.). Following the step554, processing is complete.

If it is determined at the test step 552 that the requested lease isavailable, then control transfers from the step 552 to a test step 556where it is determined if the lease request is for writing data to thefile corresponding to the lease. In an embodiment herein, multiple users(e.g., clients, servers) may read from the same file simultaneouslywhile only one user may write to the same file. Accordingly, if it isdetermined at the test step 556 that a user is not requesting writeaccess, then control transfers from the test step 556 to a step 558where the metadata servers 405 return the lease (i.e., returns anappropriate indicator/identifier corresponding to granting the lease).In an embodiment herein, leases may also be provided with apredetermined expiration time after which the leases are no longervalid. Lease expirations and lease recalls are discussed in more detailelsewhere here. In addition, leases may be provided along with securitycredentials (generated, perhaps, by the security manager servers 403)that only allow for the requested operation (e.g., read only, read andwrite, etc.). The security credentials may also expire at or around thesame time that the lease expires in order to enforce lease expirations.Following the step 558, processing is complete.

If it is determined at the test step 556 that the user is requesting awrite lease, then control transfers from the test step 558 to a teststep 562 where it is determined if another user has already obtained awrite lease for the same file. As discussed elsewhere herein, only onewrite lease at a time is granted for a file. If it is determined at thetest step 562 that another write lease has already been granted, thencontrol transfers from the test step 562 to a step 564 where a failureindicator is returned. Just as with the step 562, the failure indicatorreturned at the step 564 may include information identifying the natureof the failure. Following the step 564, processing is complete. If it isdetermined at the test step 562 that another write lease has not beengranted, then control transfers from the test step 562 to a step 566where the metadata servers 405 return the lease, possibly along with anexpiration. Following the step 566, processing is complete.

As discussed elsewhere herein, it may be desirable in some instances toissue leases with expiration dates. In an embodiment herein, aparticular one of the metadata servers 405 may be responsible for aparticular file and corresponding file objects. The responsible one ofthe metadata servers 405 issues leases for the file and correspondingfile objects and handles lease expiration processing. The leaseinformation may be stored in appropriate data structures (e.g.,table(s), linked list(s), etc.) by the responsible one of the metadataservers 405. In addition, it is possible to have more than one of themetadata servers 405 be responsible for a particular file or set offiles, where one of the metadata servers 405 is a primary server andother responsible metadata servers are secondary servers that maintainappropriate information, but do not otherwise provide services unlessthe primary server fails.

Referring to FIG. 32, a flow chart 580 illustrates steps performed by aresponsible one of the metadata servers 405 in connection withmonitoring lease expiration. Processing begins at a first step 582 wherea pointer that iterates through all of the leases (file and leasecombinations) for which the server is responsible is made to point tothe first one. Following the step 582 is a test step 584 where it isdetermined if the lease has expired. The processing performed at thestep 584 may use any appropriate technique, including comparing leaseexpiration times with the current time. If it is determined at the step584 that a lease has expired, then control transfers from the step 584to a step 586 where the lease is recalled. Recalling the lease at thestep 586 is discussed in more detail elsewhere herein.

Following the step 586, or following the step 584 if the lease has notexpired, is a step 588 where the pointer that iterates through the filesand leases for which the server is responsible is incremented. Followingthe step 588 is a test step 592 where it is determined if the pointerpoints past the end (i.e., all files and corresponding leases have beenprocessed). If so, then control transfers from the step 592 back to thestep 582, discussed above, to reset the pointer to point to the firstone and begin another pass to check for expired leases. If it isdetermined at the step 592 that the pointer does not point past the end,then control transfers from the test step 592 back to the step 584,discussed above, for another iteration.

In an embodiment herein, the system may provide close-to-openconsistency where data consistency is provided after an entity hasreleased write access. Said differently, consistency is provided for afile when no entity has an active write lease for the file. Conversely,while any entity has an active write lease, the state of the data maynot be guaranteed for any entity reading the data. In the systemdescribed herein, leases may be recalled after expiration or may berecalled for other reasons. Recalling the leases may improve theconsistency of the data being accessed by other entities.

Referring to FIG. 33, a flow chart 610 illustrates steps performed inconnection with a particular one of the metadata servers 405 recalling alease. Processing begins at a first step 612 where a message is sent tothe entity holding the lease (e.g., one of the clients 104-106) to alertthe entity that the lease is being recalled. As discussed elsewhereherein, any appropriate entity may hold a lease, including clients104-106, other servers, or even one of the metadata servers 405. In someembodiments, the step 612 may include ensuring receipt of the message bythe entity (e.g., waiting for an acknowledgement) while in otherembodiments the message is simply sent and it is assumed that themessage is received by any operational recipient. Of course, in systemsthat wait for an acknowledgement, there may be a timeout and/or a limiton the number of attempts.

Following the step 612 is a step 614 where the appropriate tables areadjusted to reflect that the lease that has been recalled is no longeroutstanding. Tables and other data structures used with leases arediscussed in more detail elsewhere herein. Following the step 614 is atest step 616 where it is determined if the lease that was recalled wasa write lease (lease to allow writing data). As discussed elsewhereherein, the system may provide close-to-open consistency so that, when awrite lease is released, data reads are made consistent. This may befacilitated by recalling all read leases when a write lease is recalled.Entities for whom a read lease is recalled may flush their internalbuffers prior to reacquiring the read lease after the recall. Note, bythe way, that an entity for which a write lease is recalled may alsoflush buffers by writing unsaved data to the physical storage 414(through the data storage servers 407) in response to receiving a recallnotification. Accordingly, in some embodiments, an entity receiving arecall message for a write lease may be provided with a certain amountof time in which to write any unsaved data to physical storage. For thispurpose, the security credentials provided along with a write lease maybe set to expire a predetermined amount of time after the write leaseexpires.

If it is determined at the step 616 that the lease that is beingrecalled is not a write lease, then processing is complete. Otherwise,control transfers from the test step 616 to a step 618 where a pointer,used to iterate through all of the outstanding leases for the file forwhich the write lease is being recalled, is made to point to the firstoutstanding lease.

Following the step 618 is a test step 622 where it is determined if thepointer points past the end (i.e., all outstanding leases have beenrecalled). If so, then processing is complete. Otherwise, controltransfers from the test step 622 to a step 624 where the lease isrecalled. The lease may be recalled by calling the processingillustrated by the flow chart 610, and thus may be recursive. Followingthe step 624 is a step 626 where the pointer used to iterate throughentities having outstanding leases for a file is incremented. Followingthe step 626, control transfers back to the step 622 for anotheriteration.

Referring to FIG. 34, a table 630 is shown as including a plurality ofentries used to keep track of files for which a particular one of themetadata servers 405 is responsible. Of course, other appropriate datastructures may be used instead of the table 630, such as a linked list,a doubly linked list, etc. Each primary one of the metadata servers 405,and any corresponding secondary one(s) of the metadata servers 405, maycontain data corresponding to specific file sets for which theparticular one(s) of the metadata servers 405 are responsible. Thus, forexample, a first primary one of the metadata servers 405 may contain aversion of the table 630 for a first set of files for which the firstprimary one of the metadata servers is responsible while a secondprimary one of the metadata servers 405 may contain a completelydifferent version of the table 630 for a second (different) set of filesfor which the second primary one of the metadata servers 405 isresponsible.

Each entry of the table 630 includes a file identifier field (FID) thatuniquely identifies the file corresponding to an entry. In an embodimentherein, the FID field may be the object id of the metadata object forthe file (for example, the metadata object 132 in the diagram 130 ofFIG. 4). Note that, as discussed elsewhere herein, the metadata objectfor a file may be used to locate all of the other data objects for thefile.

The MD LOC field may describe the data storage location of the metadataobject for the file. In an embodiment herein, the MD LOC field maycontain an identifier for the one of the data storage servers 407 thatstores the metadata object for the file. THE MD LOC field may alsocontain a unique identifier (perhaps initially assigned by the one ofthe data storage servers 407) that may be used to retrieve and storedata at the one of the data storage servers 407.

The LEASE LIST field may contain information about all entities thathave active leases outstanding for the corresponding file. In anembodiment herein, the LEASE LIST field may contain a pointer to alinked list of elements that corresponding to entities havingoutstanding leases. Of course, any other appropriate data structure(e.g., array) may be used.

Referring to FIG. 35, a diagram 640 illustrates a linked list that maybe used to keep track of entities having an outstanding active lease fora file. Each element of the list contains a NEXT field that points tothe next element in the list. The element at the end of the listcontains a null indicator. Thus, the list may be traversed by startingwith the element pointed to by the LEASE LIST pointer and subsequentlypointing to the elements pointed to by the NEXT field. (e.g., inconnection with the processing illustrated in the flow chart 580 of FIG.32). Similarly, conventional linked list operations may be used to addand remove elements.

Each element of the list also contains a TYPE field that indicates thetype of lease (e.g., read or write) and includes an EXP field thatindicates when the lease corresponding to the element expires. Eachelement also contains an ENT field that indicates the entity (e.g., oneof the clients 104-106, another server, etc.) that holds thecorresponding lease.

Manipulation of the linked list is fairly straight-forward. When a leaseis granted, fields of an element are populated with the type,expiration, and entity corresponding to the lease and the element isthen added to the list. Similarly, when a lease is recalled or otherwisereturned, the corresponding element is removed from the list. Of course,other data structures may be used instead of a linked list.

Referring to FIG. 36, a table 650 is shown as containing a plurality ofentries that correlate object identifiers (OID) with locationinformation (LOC). In an embodiment herein, object identifiers are anumerical value. Each of the metadata servers 405 is provided with aunique range of object identifiers to use/manage. In some embodiments,object identifiers may be reused while in other embodiments objectidentifiers are not reused. Of course, in instances where objectidentifiers are not reused, each of the metadata servers 405 needs to beprovided with a sufficient range of useable object identifiers.

The LOC field is like the MD LOC field for the table 630. The LOC fielddescribes the data storage location of the corresponding object. In anembodiment herein, the LOC field may contain an identifier for the oneof the data storage servers 407 containing (handling) the object as aunique identifier (perhaps assigned by the one of the data storageservers 407) that may be used to retrieve and store data for the object.Thus, if one of the metadata servers 405 has a table entry for aparticular object, an entity can pass the object identifier to the oneof the metadata servers 405 and receive in return the corresponding LOCinformation to allow the entity to access the appropriate one of datastorage servers 407 directly.

Having a number of metadata servers 405 allows for distributed servicingof file operations (and thus significant scalability) as well asproviding failover/redundancy capability. In some instances, objects maybe reassigned from one of the metadata servers 405 to another. However,since each of the metadata servers 405 contains information for only asubset of files (and corresponding file objects), it may be necessary toprovide a mechanism for locating an appropriate one of the metadataservers 405 in connection with performing operations.

The metadata location servers 408 provide location services for anentity seeking the appropriate one of the metadata servers 405 foroperations on a particular file. In an embodiment herein, each of themetadata location servers 408 may receive a call having an objectidentifier and can return a specific one of the metadata servers 405that handles the particular object. In addition, as discussed in moredetail elsewhere herein, the metadata location servers 408 may assist inconnection with the creation of new objects by indicating to a callingentity (e.g., one of the clients 104-106) a specific one of the metadataservers 405 to be used for the new object. The metadata servers 408 mayoperate like Domain Name Servers on the Web, and each of the clients104-106 (and other entities) may be provided with a primary and asecondary one of the metadata location servers 408 to consult.

Referring to FIG. 37, a table 670 is shown as containing entries for useby one of the metadata location servers 408. Each entry includes an OIDRANGE field, which indicates a range of object identifiers, and an MDSID field, which identifies a particular one of the metadata servers 405,or possibly a group of the metadata servers 405, with one being primaryand the remainder being secondary. An entity may provide a particularobject identifier to the metadata location server 408, which may thenconsult the table 670 and return the corresponding value from the MDS IDfield.

In addition, the metadata location servers 408 may assign a particularone of the metadata servers 405 in connection with creation of a newobject. The assignment may be based on any appropriate metric, includingrandom assignment, assignment based on geographic proximity, loadbalancing, and/or a policy input by a user through the user managementinterface 412, discussed above. A policy may indicate, for example, thatnew objects created by a particular client are provided on a particularmetadata server.

Referring to FIG. 38, a flow chart 680 illustrates processing by one ofthe metadata location servers 408 to assign a particular one of themetadata servers in connection with creation of a new file object.Processing begins at a first step 682 where the policy may be consulted.As discussed elsewhere herein, it may be possible to input policiesthrough the user management interface 412. The policies may dictate(directly or indirectly) which of the metadata servers 405 are to beused for which of the clients 104-106. Note that other policies arepossible. For example, the policies may indicate which of the metadataservers 405 are to be used at different times of the day (independent ofthe clients 104-106) or based on the load, the user, etc. The policyinformation may be stored at the user management interface 412 andaccessed in connection with the step 682, or may be stored at the one ofthe metadata location servers 408 after having been previously passedthereto. Following the step 682 is a step 684 where identificationinformation for a specific one of the metadata location servers 405 isreturned to the calling entity. Following the step 684, processing iscomplete.

Referring to FIG. 39, a flow chart 700 illustrates steps performed byone of the metadata servers 405 in connection with deleting a file forwhich the one of the metadata servers 405 is responsible. Processingbegins at a first step 702 where all leases for the file are recalled(e.g., by iterating through the lease list 640 and providing recallprocessing like that illustrated by the flow chart 610 of FIG. 33).Following the step 702 is a step 704 where the leases are madeunavailable for other processes (e.g., by setting an appropriate flagthat prevents granting further leases for the file). Following the step704 is a step 706 where the metadata for the file is obtained (e.g., byretrieving the metadata object for the file). Note that, as discussedelsewhere herein, the metadata object for a file contains informationabout the location of all the file objects used in connection with thefile.

Following the step 706 is a step 708 where a pointer, used to iteratethrough the objects used in connection with a file, is made to point tothe first object. The pointer and subsequent iterative processing usesinformation obtained at the step 706 to determine the objects for thefile. Following the step 708 is a test step 712 where it is determinedif the pointer, used to iterate through the objects, points past the end(i.e., all of the file objects have been processed). If so, then controltransfers from the test step 712 to a step 714 where the table entrycorresponding to the file (i.e., the entry in the table 630 of FIG. 34)is deleted (e.g., set to null). Following the step 714, processing iscomplete.

If it is determined at the step 712 that there are more file objects toprocess, then control transfers from the step 712 to a step 716 wherethe LOC information is obtained for the object. The LOC information islike the information stored in the table 650 of FIG. 36, discussedabove. In some instances, the LOC information will be local to the oneof the metadata servers 405 performing the processing. In otherinstances, it may be necessary to call one of the metadata locationservers 408 to get the location information for the object. Followingthe step 716 is a step 718 where a message is sent to the appropriateone of the data storage servers 407 (i.e., the one handling the object)to cause the object to be deleted.

In an embodiment herein, it may be possible for different files to usethe same object (e.g., deduplication, file aliasing, etc.), in whichcase the one of the data storage servers 407 would simply decrement acounter for the object indicating the number of users thereof. When thecounter is decremented to zero, the data storage server may delete thedata corresponding to the object. Note that the object(s) associatedwith a file may be deleted asynchronously. Following the step 718 is astep 722 where the pointer used to iterate through the file objects isincremented. Following the step 722, control transfers back to the step712, discussed above, for another iteration.

Referring to FIG. 40, a flow chart 740 illustrates steps performed byone of the metadata servers 405 in connection with creating a new file.Note that, prior to performing the processing illustrated in FIG. 40,the entity creating the file (e.g., one of the clients 104-106) mayfirst consult the metadata location servers 408 to determine the properone of the metadata servers 405 to use to create the file.

Processing begins at a first step 742 where the policy manager servers402 are consulted to obtain policy information for new files (e.g., newfiles for client X have a mirror geographically located at least acertain distance from the primary data set). Following the step 742 is astep 744 where the resource manager servers 406 are consulted todetermine the available resources to meet the dictates of the policyobtained at the step 742. Following the step 744 is a test step 746where it is determined if it is possible to meet the dictates of thepolicy given the available resources. For example, it may not bepossible to satisfy the policy of having geographically separatedmirrors if all of the remaining physical storage in a system is in onegeographic location. If it is determined at the test step 746 that it isnot possible to fulfill a policy, then control transfers from the teststep 746 to a step 748 where alternative processing is performed. Anyappropriate processing may be performed at the step 748, includingreturning an error indicator to the calling entity, creating the filewith the next best available resources, etc. Following the step 748,processing is complete.

If it is determined at the step 746 that it is possible to fulfill thepolicy with available resources, then control transfers from the teststep 746 to a step 752 where the metadata object for the file iscreated. Creating the metadata object may include populating the datafields of the metadata object and obtaining storage from an appropriateone of the data storage servers 407. In an embodiment herein, the datastorage servers 407 may be a pool and, absent any other specificrequirements, may provide storage space at any appropriate portion ofthe physical storage 414 upon request. The metadata objects created atthe step 752 will be like those described herein. See, for example,FIGS. 4-8 and the corresponding description. Following the step 752 is astep 754 where a table entry is created in the table 630 for the newfile. Following the step 754 is a step 756 where the object id of themetadata object for the file is returned to the calling entity.Following the step 756, processing is complete.

As discussed elsewhere herein, when a client or other entityunsuccessfully attempts a write operation, a message (update) is sent tothe servers 102 by the client or other entity. Similarly, a message(update) may also be sent to the servers 102 in connection with findinga stale mirror in connection with a synchronous mirror copy (see thestep 358 of the flow chart 350 of FIG. 13), and/or writing to datahaving an asynchronous mirror.

Referring to FIG. 41, a flow chart 760 illustrates steps performed byone of the metadata servers 405 in connection with handling a messagethat a write operation was unsuccessful. Processing begins at a firststep 762 where all of the leases for the file are recalled. Followingthe step 762 is a step 764 where leases for the file are madeunavailable. Following the step 764 is a step 766 where new storagespace is allocated to replace the old storage space to which the writeoperation was unsuccessful. Following the step 766 is a step 768 wherethe appropriate information in the metadata object for the file isadjusted. Following the step 768 is a step 772 where a data copyoperation is begun to repopulate the new data storage space with, forexample, data from one of the mirrors. While the data copy operation isbeing performed, the data may be indicated as being stale at the step772. Of course, if there are no mirrors or other data that can be usedto repopulate the new storage space, then the processing at the step 772would not be performed. Following the step 772 is a step 774 where theleases for the file are made available. Following the step 774,processing is complete.

Referring to FIG. 42, a flow chart 780 illustrates processing preformedin connection with one of the metadata servers 405 receiving anindication that a synchronous mirror was stale. Processing begins at afirst test step 782 where it is determined if the mirror is currently inthe process of being populated with data (perhaps in connection with aprevious bad write operation). If so, then the data population operationis allowed to continue and processing is complete. Otherwise, controltransfers from the test step 782 to a step 784 where processing likethat illustrated in the flow chart 760 of FIG. 41, discussed above, isperformed. Following the step 784, processing is complete.

Referring to FIG. 43, a flow chart 790 illustrates steps performed inconnection with the servers 102 receiving a message that datacorresponding to asynchronous copies has been written, thusnecessitating update of the asynchronous copies. Processing begins at afirst step 792 where information is added to a queue (e.g., a job queueprovided at the affected one of the metadata servers 405) indicatingthat asynchronous data needs to be copied. As discussed in moreelsewhere herein, a process at each of the metadata servers 405 servicesthe corresponding queue. Following the step 792, processing is complete.

Referring to FIG. 44, a flow chart 800 illustrates processing performedby a process at each of the metadata servers 405 that services thecorresponding queue that is populated by the processing illustrated bythe flow chart 790 of FIG. 43. Processing begins at a first test step802 where it is determined if the queue is empty. If so, then controltransfers back to the test step 802 to continue to poll the queue. Ifthe queue is not empty, then control transfers from the test step 802 toa step 804 where the next entry in the queue is removed. In anembodiment herein, queue elements may be processed on a first in firstout basis. However, it is also possible to selectively remove elementsfrom the queue in any order. For example, elements associated with filesthat receive higher priority may be removed before other elements (e.g.,in connection with a flush operation for the file). As another example,elements may be removed according to size (of needed write operation) oraccording to any other appropriate criteria.

Following the step 804 is a step 806 where the write lease for theaffected file is recalled. Following the step 806 is a step 808 wherethe write lease is obtained. Following the step 808 is a step 812 wherethe write operation is performed to write the asynchronous data to themirror. Following the step 812 is a step 814 where the write lock isreleased. Following the step 814, control transfers back to the step 802to continue to poll the queue.

The system described herein may be used with any server, or any group ofservers, capable of providing file objects to clients. The particularform of the file objects may vary without departing from the spirit andscope of the invention. In some instances, the order of steps in theflow charts may be modified, where appropriate. The system describedherein may be implemented using a computer program product/softwareprovided in a computer-readable storage medium.

While the invention has been disclosed in connection with variousembodiments, modifications thereon will be readily apparent to thoseskilled in the art. Accordingly, the spirit and scope of the inventionis set forth in the following claims.

What is claimed is:
 1. A method of managing file objects in a datastorage system, comprising: providing a plurality of metadata servers,each metadata server having information for only a subset of fileobjects in the storage system, each subset for each of the metadataservers being less than a total number of file objects for the system;and providing at least one metadata location server, different from themetadata servers, wherein an entity accessing a specific file object ofthe system determines which of the metadata servers contains data forthe specific file object by first accessing the at least one metadatalocation server; wherein the metadata location server assigns aparticular one of the metadata servers in connection with creation of anew file object.
 2. The method, according to claim 1, wherein theinformation for the subset of file objects includes a table having fileidentifiers and metadata location information.
 3. The method, accordingto claim 2, wherein the metadata location information points to datastorage for a metadata object for a file.
 4. The method, according toclaim 2, wherein the table also includes lease information thatindicates entities that have read permission for the file object andentities that have write permission for the file object.
 5. The method,according to claim 1, wherein the file objects are provided with uniqueobject identifiers.
 6. The method, according to claim 5, wherein the atleast one metadata location server determines an appropriate metadataserver to use for a file object according to an object identificationvalue for the file object.
 7. The method, according to claim 5, whereinat least some of the metadata servers maintain a table that correlates asubset of the object identification values with file objects.
 8. Themethod, according to claim 7, wherein at least some of the metadataservers are provided with a unique subset of object identificationvalues to assign to new file objects.
 9. The method, according to claim1, wherein assignment of the particular one of the metadata servers israndom or is based on at least one of: geographic proximity, loadbalancing, and a policy.
 10. A Computer software, provided in anon-transitory computer-readable medium, that manages file objects in adata storage system, the software comprising: executable code, providedon a plurality of metadata servers, that, when executed by a processor,maintains information for only a subset of file objects in the storagesystem, each subset for each of the metadata servers being less than atotal number of file objects for the system; and executable code,provided on at least one metadata location server different from themetadata servers, that, when executed by a processor, provides anindication of which of the metadata servers contains data for a specificfile object in response to a request therefor provided to the at leastone metadata server; wherein the metadata location server assigns aparticular one of the metadata servers in connection with creation of anew file object.
 11. The Computer software, according to claim 10,wherein the information for the subset of file objects includes a tablehaving file identifiers and metadata location information.
 12. TheComputer software, according to claim 11, wherein the metadata locationinformation points to data storage for a metadata object for a file. 13.The Computer software, according to claim 11, wherein the table alsoincludes lease information that indicates entities that have readpermission for the file object and entities that have write permissionfor the file object.
 14. The Computer software, according to claim 10,wherein the file objects are provided with unique object identifiers.15. The Computer software, according to claim 14, wherein the at leastone metadata location server includes executable code that determines anappropriate metadata server to use for a file object according to anobject identification value for the file object.
 16. The Computersoftware, according to claim 14, wherein at least some of the metadataservers maintain a table that correlates a subset of the objectidentification values with file objects.
 17. The Computer software,according to claim 16, wherein at least some of the metadata servers areprovided with a unique subset of object identification values to assignto new file objects.
 18. The Computer software, according to claim 10,wherein assignment of the particular one of the metadata servers israndom or is based on at least one of: geographic proximity, loadbalancing, and a policy.
 19. A data storage system, comprising: aplurality of clients that access file objects of the storage system; anda plurality of interconnected servers coupled to the clients, theservers including a plurality of metadata servers, each metadata serverhaving information for only a subset of file objects in the storagesystem, each subset for each of the metadata servers being less than atotal number of file objects for the system, the servers also includingat least one metadata location server different from the metadataservers, wherein a client accessing a specific file object of the systemdetermines which of the metadata servers contains data for the specificfile object by first accessing the at least one metadata locationserver; wherein the at least one metadata location server assigns aparticular one of the metadata servers in connection with creation of anew file object.
 20. The data storage system, according to claim 19,wherein the information for the subset of file objects includes a tablehaving file identifiers and metadata location information.
 21. The datastorage system, according to claim 20, wherein the metadata locationinformation points to data storage for a metadata object for a file. 22.The data storage system, according to claim 19, wherein the file objectsare provided with unique object identifiers.
 23. The data storagesystem, according to claim 22, wherein the at least one metadatalocation server determines an appropriate metadata server to use for afile object according to an object identification value for the fileobjects for the file.
 24. The data storage system, according to claim22, wherein at least some of the metadata servers maintain a table thatcorrelates a subset of the object identification values with fileobjects.
 25. The data storage system, according to claim 24, wherein atleast some of the metadata servers are provided with a unique subset ofobject identification values to assign to new file objects.
 26. The datastorage system, according to claim 19, wherein assignment of theparticular one of the metadata servers is random or is based on at leastone of: geographic proximity, load balancing, and a policy.